How to **include** files

General Topics
1

Introduction

There are a few questions on how to include files (not just exclude), and a few answers. I’ve solved this for me, and wanted to document it here in case this helps others. I’ll start with an example .kopiaignore but also show how to test things quickly for your own use case.

Example .kopiaignore to include some files and folders

The following .kopiaignore in my home folder does the following:

  • Ignores all folders named G or G_* anywhere including sub-folders
  • Ignores all tmp folders
  • Ignores the data, ESSD, and other folder
  • Includes all files named YES.file or *.org everywhere, even if they are in excluded folders
  • Includes all PDF files in other, but not elsewhere
  • Includes all folders named 2023 and everything under them (even files named NO.file), even if they are previously excluded.
  • You can add more exclusions after the inclusions, for example to skip all NO.file, even though the 2023 line includes them.
G
G_*
tmp
data/**
ESSD/**
other/**
!**/
!**/**/*.org
!**/**/YES.file
!other/**/*.pdf
!**/**/2023/**

How to test your own exclude/include filter

Make test directory structure

# Set up synthetic files and folders in ~/tmp/foo
mkdir -p tmp/foo
cd tmp/foo

# Set up folder structure and touch files
mkdir -p ESSD/{2023,2024}/{a,b,c}/data/{v1,v2}
touch ESSD/{2023,2024}/{a,b,c}/data/{v1,v2}/NO.file
touch ESSD/{2023,2024}/{a,b,c}/YES.file

mkdir -p data/{foo,bar,baz}/{a,b,c}/{one,two,three}
touch data/{foo,bar,baz}/{a,b,c}/readme.org
touch data/{foo,bar,baz}/readme.org
touch data/{foo,bar,baz}/{a,b,c}/NO.dat

mkdir -p other/{tmp,foo}/{tmp,bar}/tmp
touch other/{tmp,foo}/{tmp,bar}/tmp/YES.file
touch other/{tmp,foo}/{tmp,bar}/YES.file
touch other/{tmp,foo}/YES.file
touch other/YES.file
mkdir -p other/{tmp,foo}/{tmp,bar}/{G,G_foo}
mkdir -p other/{tmp,foo}/{G,G_foo}

touch other/tmp/TEST.pdf data/foo/TEST.pdf

After the above commands, I have the following folder and file structure in ~/tmp/foo/

$ tree
.
├── data
│   ├── bar
│   │   ├── a
│   │   │   ├── NO.dat
│   │   │   ├── one
│   │   │   ├── readme.org
│   │   │   ├── three
│   │   │   └── two
│   │   ├── b
│   │   │   ├── NO.dat
│   │   │   ├── one
│   │   │   ├── readme.org
│   │   │   ├── three
│   │   │   └── two
│   │   ├── c
│   │   │   ├── NO.dat
│   │   │   ├── one
│   │   │   ├── readme.org
│   │   │   ├── three
│   │   │   └── two
│   │   └── readme.org
│   ├── baz
│   │   ├── a
│   │   │   ├── NO.dat
│   │   │   ├── one
│   │   │   ├── readme.org
│   │   │   ├── three
│   │   │   └── two
│   │   ├── b
│   │   │   ├── NO.dat
│   │   │   ├── one
│   │   │   ├── readme.org
│   │   │   ├── three
│   │   │   └── two
│   │   ├── c
│   │   │   ├── NO.dat
│   │   │   ├── one
│   │   │   ├── readme.org
│   │   │   ├── three
│   │   │   └── two
│   │   └── readme.org
│   └── foo
│       ├── a
│       │   ├── NO.dat
│       │   ├── one
│       │   ├── readme.org
│       │   ├── three
│       │   └── two
│       ├── b
│       │   ├── NO.dat
│       │   ├── one
│       │   ├── readme.org
│       │   ├── three
│       │   └── two
│       ├── c
│       │   ├── NO.dat
│       │   ├── one
│       │   ├── readme.org
│       │   ├── three
│       │   └── two
│       ├── readme.org
│       └── TEST.pdf
├── ESSD
│   ├── 2023
│   │   ├── a
│   │   │   ├── data
│   │   │   │   ├── v1
│   │   │   │   │   └── NO.file
│   │   │   │   └── v2
│   │   │   │       └── NO.file
│   │   │   └── YES.file
│   │   ├── b
│   │   │   ├── data
│   │   │   │   ├── v1
│   │   │   │   │   └── NO.file
│   │   │   │   └── v2
│   │   │   │       └── NO.file
│   │   │   └── YES.file
│   │   └── c
│   │       ├── data
│   │       │   ├── v1
│   │       │   │   └── NO.file
│   │       │   └── v2
│   │       │       └── NO.file
│   │       └── YES.file
│   └── 2024
│       ├── a
│       │   ├── data
│       │   │   ├── v1
│       │   │   │   └── NO.file
│       │   │   └── v2
│       │   │       └── NO.file
│       │   └── YES.file
│       ├── b
│       │   ├── data
│       │   │   ├── v1
│       │   │   │   └── NO.file
│       │   │   └── v2
│       │   │       └── NO.file
│       │   └── YES.file
│       └── c
│           ├── data
│           │   ├── v1
│           │   │   └── NO.file
│           │   └── v2
│           │       └── NO.file
│           └── YES.file
└── other
    ├── foo
    │   ├── bar
    │   │   ├── G
    │   │   ├── G_foo
    │   │   ├── tmp
    │   │   │   └── YES.file
    │   │   └── YES.file
    │   ├── G
    │   ├── G_foo
    │   ├── tmp
    │   │   ├── G
    │   │   ├── G_foo
    │   │   ├── tmp
    │   │   │   └── YES.file
    │   │   └── YES.file
    │   └── YES.file
    ├── tmp
    │   ├── bar
    │   │   ├── G
    │   │   ├── G_foo
    │   │   ├── tmp
    │   │   │   └── YES.file
    │   │   └── YES.file
    │   ├── G
    │   ├── G_foo
    │   ├── TEST.pdf
    │   ├── tmp
    │   │   ├── G
    │   │   ├── G_foo
    │   │   ├── tmp
    │   │   │   └── YES.file
    │   │   └── YES.file
    │   └── YES.file
    └── YES.file

90 directories, 52 files

Back up files and then immediately list files that were backed up

Set up Kopia (in GUI or however you prefer) to snapshot your ~/tmp/foo folder one time. Dont’ worry about ignoring files for this first backup. We’ll do further backups and comparisons on the command line.

The following script performs a backup, mounts it, lists the files, and then unmounts it. I’m running this in Org Babel code blocks so it’s easy to tweak the HEREDOC section and then just rerun to regenerate the output.

# Recreate .kopiaignore
cat << EOF >> ~/tmp/foo/.kopiaignore
G
G_*
tmp
data/**
ESSD/**
other/**
!**/
!**/**/*.org
!**/**/2023/**
!**/**/YES
!other/**/*.pdf
EOF

kopia snapshot create /home/kdm/tmp/foo/ # take a snapshot

# find latest
id=$(kopia snapshot list /home/kdm/tmp/foo/ | grep latest-1 | tail -n1 | cut -d" " -f6) && echo $id
mntdir="/tmp/kopia-mnt"
mkdir ${mntdir} # only run 1x, then comment out

kopia mount ${id} ${mntdir} & # mount latest in background but wait for mount to finish
sleep 1

find ${mntdir} # list files

kill %1 # kill mount

When I run the above code, I immediately get a file list. I can easily tweak the .kopiaignore and re-run to test patterns and exclude-then-include the specific folders, files, and file types that I want.

Clean up

  • Move your .kopiaignore to your real (big) backup
  • Delete your test folder on disk
  • Delete your snapshots in Kopia

I hope this is helpful to others.

3 Likes