File inclusion seems to fail, no matter how I try to configure it

[IPlayZed]

Hi!

I am trying to back up my home directory.
I am using systemd-homed, which uses the following structure in /home:
/home
/home/user.home (this is the image file which is mounted when the user is authenticated)
/home/user (this is the temporary mount of /home/user.home, which is empty when the user is not authenticated and is only readable for the user)
I am using a systemd service to initiate the Kopia backup for /home.
This all means that I want to backup each .home file in /home, but ignore the content or the created mount folder of each mounted directory.

So in my example /home/user.home would be backed up, but /home/user’s contents would not be attempted to be backup up, so ignored. This would be true for user2, user3 and so on.

Now, I looked at the docs and forum posts, but the solutions I have tried or were posted there, seems to not backup anything.

I have created the following ignore policy for /home:
Files policy:
Ignore cache directories: true inherited from (global)
Ignore rules: (defined for this target)
!/*.home
*
Read ignore rules from files: inherited from (global)
.kopiaignore

As far as I understand, this should ignore everything, but the .home files in /home.

Then I call: sudo kopia snapshot create /home (for now root is the user doing the backup, maybe I will make an unprivilaged readonly user which can access everything):

Snapshotting root@minefpc:/home …

• 0 hashing, 0 hashed (0 B), 0 cached (0 B), uploaded 0 B, estimating…
  Created snapshot with root k7f976c0894948354dcee4943670fda05 and ID e471effae9701b1fae2f527ed1f8d23f in 0s

So nothing…

What am I missing? I feel like I am going crazy, this should be such a simple thing to do.

Relevant posts:

How to include only some files from folder?

tthis /* !/.var/app/com.nextcloud.desktopclient.nextcloud/config/Nextcloud/nextcloud.cfg produces empty snapshot. this /* !/armbian/ /armbian/* !/armbian/userpatches !/armbian/.git/ /armbian/.git/* !/armbian/.git/config work as I need: I got in snapshot file armbian/.git/config and armbian/userpatches folder with content. Then, the border is more than one folder deep for exclusion: if path is “a/b/c”, then it’s possible to exclude “a” but get back “b/c”. but impossible to exclude complete…

Include vs Exclude in .kopiaignore

OK, had a chance to test more. Here are my findings with Version v20210302.0.82937 built on Tuesday, March 2, 2021 4:36:51 PM fv-az15-142 using the “.kopiaignore” file. I can get it to exclude everything except certain file types if I also explicitly include each directory that makes up the path to the files I want included. For example, if I want .ai files in the path /a/b/c/d I need to do: * !a/ !a/b/ !a/b/c/ !a/b/c/d/ !*.ai However, I want it to match .ai files for all 100’s of subfolders…

[IPlayZed]

As a stopby solution, I manually specified the folders to be excluded, but that is not viable when users could dynamically be added or removed.

[kapitainsky]

Have a look at this post. Maybe it will give you some clue how to proceed.

Backup Exclusion Issue: Specific Directories root drive

I’m having an issue with setting up exclusions in my backup configuration and would appreciate any assistance. My root directory for the backup is located at D:\ I want to exclude everything from the backup except for the following directories and their subdirectories: D:\Backup\OLD D:\Sys\Docs D:\Data\bd There are multiple files and folders at the root of D:\, as well as within subdirectories, that I want to exclude. For example, I want to exclude files like D:\Backup\list.txt and folders …

At the end it is just about using right exclusion/inclusion patterns.

[IPlayZed]

Hi, thanks for the answer.

So I took a look at the post you linked the only difference I see is that you used the exclude-all pattern only on the root of the backup, so /* instead of *.

I changed my policy to:
Files policy:
Ignore cache directories: true inherited from (global)
Ignore rules: (defined for this target)
!/*.home
/*
I do not see how this would affect btw anything, because there this only means that by default not all, but patterns only in the root should be ignored, but I changed it to your suggestion.

However, the problem persists still.

I even tried specifying !/admin.home, but it is still not being backed up.

It is like if Kopia did not match files with the pattern /something.home.

[kapitainsky]

@IPlayZed

This all means that I want to backup each .home file in /home, but ignore the content or the created mount folder of each mounted directory.

If too complex to create exclusions maybe set --one-file-system policy. It will skip all mounts.

[IPlayZed]

Multiple problems with this:

• This is a non-solution, even if it would work. It does not solve the underlying problem. I encourage you to try to replicate my directory structure and check if it works on your machine, if you are unsure what the problem is and are interested enough to help.
• The other issue is that homed home directories are loopback devices. This means that they are still on the same filesystem, but just looped to a file.
• It would not be a general solution. What if I have a similar structure, but not for homed stuff? I am still stuck.

If you think this is a bug, I can open a GH issue.

[kapitainsky]

I do not think there is any bug in exclusion logic. It is just a bit tricky to get it right first time. I have no time now to replicate your issue in details.

@IPlayZed

!/*.home
/*

kopia just follows your instructions - what you ask for is to ignore all but /*.home and then to ignore all /* - so correctly nothing is backed up.

Try it other way around - first exclude all and then reverse it for specific items.

[IPlayZed]

I see, I did not know that order matters. When using kopia policy set --add-ignore=..., in which order does it save it? Is there a way to manually edit this policy?

[kapitainsky]

@IPlayZed

Is there a way to manually edit this policy?

Use kopiaUI…

if no success please post your /home structure - not description but something like $ tree /home or similar with key elements and what is your expectation in terms of applied rule results (in similar format).

I will then try to replicate it and create required rule in the next few days - sorry but do not have right device now (ipad is not good for that).

[IPlayZed]

I can’t use KopiaUI, because the user is a headless one (either root, or a specific service user), not a human-user. Also, I would like to be able to do this in the cli.
The tree output:

admin@minefpc ~ [2]> tree /home -L 2
/home
├── admin
│ ├── Desktop
│ ├── desktop-backup.fd.sd.tar.gz
│ ├── Documents
│ ├── Downloads
│ ├── go
│ ├── kernel
│ ├── kernel-build
│ ├── kopia-config.txt
│ ├── Music
│ ├── pacman-hooks → /etc/pacman.d/hooks
│ ├── Pictures
│ ├── Public
│ ├── snap
│ ├── Templates
│ ├── usr-local-bin → /usr/local/bin/
│ └── Videos
├── admin.home
├── gaming.home
├── personal [error opening dir]
└── personal.home

(Kinda offtopic:You can also see here btw. how systemd-homed separates the user dir personal, so no other user, even root would be able to peek in it .)
I do not think that the policy order should matter, gitignore doesn’t care about order, so this should not as well AFAIK, am I wrong there?

Also, I tried issuing the commands in both orders:

1. sudo kopia policy set --add-ignore "/*" /home
2. sudo kopia policy set --add-ignore "!/*.home" /home
   and
3. sudo kopia policy set --add-ignore "!/*.home" /home
4. sudo kopia policy set --add-ignore "/*" /home

In both cases sudo kopia policy show /home produces the same output:

Ignore rules: (defined for this target)
!/.home
/

I think it just prints the ignore rules in alphabetical order…
Anyway, thanks for helping and putting up with me. I will stick for the time being to manually specifying folders, but it would be awesome if you as a more experienced Kopia user could figure out what is happening here. Or someone else, I am really out of ideas.

[kapitainsky]

@IPlayZed

I can’t use KopiaUI,

You can connect remotely to the UI

something like this

GUI client for a remote ubuntu server

I would like to use Kopia to backup my ubuntu server running on Rackspace. The server doesn’t have a gui interface but is there a way of using ssh or similar to connect a gui interface on my laptop to control backups on the ubuntu server? Thanks in advance.

@IPlayZed

gitignore doesn’t care about order,

where did you find that kopia is using .gitignore logic?

[IPlayZed]

I still would like to solve it without involving UI, but thanks for the tip.

.kopiaignore file usage

The format is meant to be (a subset of) .gitignore. There are some things that don’t work from that spec (https://git-scm.com/docs/gitignore) most notably absolute paths. There are currently issues filed on GitHub to get it fixed. Any help here is appreciated.

[kapitainsky]

@IPlayZed

I still would like to solve it without involving UI

Here you are, test data:

$ tree  /Users/kptsky/Temp/test1
/Users/kptsky/Temp/test1
├── user1
│   └── test
├── user1.home
├── user2
│   └── test1
└── user2.home

3 directories, 4 files

with ignore patterns:

/*
!/*.home

$ kopia policy show /Users/kptsky/Temp/test1
...
Files policy:
  Ignore cache directories:          true   inherited from (global)
  Ignore rules:                             (defined for this target)
    /*
    !/*.home
  Read ignore rules from files:             inherited from (global)
    .kopiaignore
  Scan one filesystem only:         false   inherited from (global)
...

now backup:

kopia snapshot create /Users/kptsky/Temp/test1

result:

$ kopia ls -l k4b592f2c3555cabd5ea6373f67ced2f3
-rw-r--r--            6 2024-02-24 08:19:42 GMT e54f1cf2ca56a2f26f2351e27dd77bf9   user1.home
-rw-r--r--            7 2024-02-24 08:20:02 GMT b63251c6810df3815fef3e4e5aae1bad   user2.home

for ignore patterns

!/*.home
/*

as expected snapshot does not contain any data as all is excluded.

@IPlayZed

Also, I tried issuing the commands in both orders:

1. sudo kopia policy set --add-ignore "/*" /home
2. sudo kopia policy set --add-ignore "!/*.home" /home
   and
3. sudo kopia policy set --add-ignore "!/*.home" /home
4. sudo kopia policy set --add-ignore "/*" /home

In both cases sudo kopia policy show /home produces the same output:

Maybe kopia policy set --add-ignore limitation… maybe not documented clearly… so do not use it.

Run:

kopia policy edit /home

and arrange order manually:

  # Which files to include in snapshots. Options include:
  #   "ignore": ["*.ext", "*.ext2"]
  #   "ignoreDotFiles": [".gitignore", ".kopiaignore"]
  #   "maxFileSize": number
  #   "noParentDotFiles": true
  #   "noParentIgnore": true
  #   "oneFileSystem": false
  "files": {
    "ignore": [
      "/*",
      "!/*.home"
    ]
  },
  "errorHandling": {},

@IPlayZed

I do not think that the policy order should matter, gitignore doesn’t care about order

Negation pattern is a bit different - from man gitignore page:

An optional prefix “!” which negates the pattern; any matching file excluded by a previous pattern will become included again.

The word “previous” clearly implies order…

[IPlayZed]

Thanks so much!
It is partly me to blame, because I did not know there is kopia policy edit and I clearly skipped over the part explaining the order’s importance.
It seems like this worthy a GH issue, because if kopia policy set --add-ignore does not care about the order the command is issued or there is no parameter the define the priority, it makes it impossible to create some policies with it. There is also no indication of it in the manual. I will create one.

[kapitainsky]

@IPlayZed

It is partly me to blame

Not really. I was for a moment confused too. Especially that I use UI for many things - and then it works out of the box.

@IPlayZed

I will create one.

This is the right attitude. And only way to make documentation better.

[IPlayZed]

@kapitainsky it also looks like this created an unexpected bug(?) as well:

I am also snapshotting the real root directory of the system, separately from snapshotting the /home directory of the system.
In the ignore there is /home/*. This added the global /home directory to the snapshot, but not the files.
Now, after applying the policy we discussed to snapshotting /home the /home directory’s files are redundantly added to the root’s (so /'s) snapshot, unless I specify /home for the real root’s ignore.
Any idea why is this? This results in no /home directory being created for the system’s root. This is easily solvable when restoring by manually adding the /home directory, but this is incorrect…

[kapitainsky]

@IPlayZed

Any idea why is this?

Unfortunately no idea:) Not sure if it is by design or by mistake. I do not understand really how rules interact in such situations. I have noticed it myself and as a result I just backup one path only (e.g. /) and use one set of rules to control it.

It would require kopia creator to document it in more details. Or somebody to try to understand logic from the source code.

You could try to use .kopiaignore files maybe instead?

[IPlayZed]

I see, I will file a bug for this as well.