Trying and failing to use Kopia-ui to do a local backup

Support
1

Hi folk, I’m trying to use Kopia to take a manual snapshot of my Ubuntu 24.04 system. This is a local snapshot. I have a RAID0 pair with an lvm filesystem which I want to snapshot, and I’m trying to use a repo created in another filesystem on a RAID1 pair, which is mounted in the lvm filesystem. I have modified the global policy to exclude the mount point. The repo appears to have been created OK.

However, there’s something I’m not getting my head around. When Kopia prompts me for what path to snapshot, I give it ‘/’, because I want to snapshot everything in that filesystem starting at root. When I hit ‘estimate’, it fails with some problem accessing ‘/sys’ files that it can’t read or aren’t there, but tells me that it’s backing up some implausibly small number of files that just looks like maybe it’s the content of ‘/’ rather than everything underneath ‘/’.

What do I need to say to the ‘what path are you snapshotting’ prompt in order for it do the whole filesystem ?

2

Ah, OK, now I see it’s at least partly a capabilities issue. I’ve had to fix a few things there because of 24.04’s stricter take on capabilities. Though I wouldn’t expect to have to run as root or fiddle with capabilities for it to be able to read the stuff my user can read, which is all I really need it to do. I wasn’t expecting it to back up other users private stuff - there aren’t any, apart from root. Odd.

Anyway, this post ( Kopia UI as root - #12 by kopund ) was helpful.

However, with capabilities tweaked, doing an Estimate gives me a different set of errors :slight_smile: :

Blockquote
FAILED Bytes: 140.7 TB (0 B excluded) Files: 103936 (0 excluded) Directories: 19141 (0 excluded) Errors: 6 (0 ignored)
02:18:37.955 error in proc/1/task/1/fdinfo: unable to read directory: open //proc/1/task/1/fdinfo: permission denied
02:18:37.955 error in proc/1/task/1: unable to read directory: open //proc/1/task/1/fdinfo: permission denied
02:18:37.955 error in proc/1/task: unable to read directory: open //proc/1/task/1/fdinfo: permission denied
02:18:37.955 error in proc/1: unable to read directory: open //proc/1/task/1/fdinfo: permission denied
02:18:37.955 error in proc: unable to read directory: open //proc/1/task/1/fdinfo: permission denied
02:18:37.955 error in .: unable to read directory: open //proc/1/task/1/fdinfo: permission denied
02:18:37.956 Included files 100 TB… 1 TB: 1 files, total size 140.7 TB
02:18:37.956 Included files 100 MB… 1 GB: 4 files, total size 1.1 GB
02:18:37.956 Included files 10 MB …100 MB: 7 files, total size 178.3 MB
02:18:37.956 Included files 1 MB … 10 MB: 18 files, total size 46.4 MB
02:18:37.956 Included files 100 KB… 1 MB: 173 files, total size 41.6 MB
02:18:37.956 Included files 10 KB …100 KB: 1222 files, total size 37.7 MB
02:18:37.956 Included files 1 KB … 10 KB: 53705 files, total size 216.1 MB
02:18:37.956 Included files 0 B … 1 KB: 48806 files, total size 1.7 MB
02:18:37.956 Excluded files: None

I do not, in fact, possess 140.7 TB of storage. So it’s still unclear what information is actually being provided here, but, the number of files and directories is now at least plausible.

When I start the snapshot at ‘/’, it still insists that it’s backing up 140.7 TB, and the ‘details’ view shows nonsensical numbers for ‘processed bytes’ and hashed bytes/files by comparison with what df -k is telling me, even though the list of files being processed is sensible. It also lists 18446 errors, and occasionally appears to throw a hex dump into the log, though it immediately disappears again.

endstate
endstate1010×1116 152 KB

Eventually it gets to 138.6Gb uploaded, and stops processing files, though it keeps on processing bytes for some reason. Several minutes later, I hit stop. It sits there saying ‘Canceling’ for a very long time. After 15 minutes of cancelling, I terminate it.

Overall I’m afraid this is all rather unimpressive :(.

3

If you want to snapshot your complete install, you’ll have to exlude at least these dirs:

/sys, /proc, /dev since they’re volatile and created on boot. They only hold either temporary or virtual files. You may also exclude /tmp ans /run, depending on your Linux distro.

Also, as you already mentioned, you’d want to exclude your repo path as well.

4

Cool, thank you, that has helped.

I have purged and reinstalled kopia-ui, reapplied the capabilities tweak, deleted and re-created the original repo. Incidentally, the purge ( i.e. apt purge ) did not remove the kopia config from .config as I was expecting, so initially kopia failed to come up as it was looking for a non existent repo ).

I knew /proc was not a real filesystem but didn’t appreciate it was 140 TB of ‘not real’ ! So I’ve added /proc, /run, and /sys to the ignore list. I’ve also checked the tickbox for ‘only one filesystem’, and added 5% parity. Now, when I say snapshot ‘/’, the estimate tells me it will backup 179 GB.

However, I still can’t explain where this is coming from. The numbers in the ‘included files’ report add up correctly ( i.e. to 179 GB ). But df -k is reporting the size of the root filesystem as 150 GB.

Filesystem                    1K-blocks      Used  Available Use% Mounted on
tmpfs                           4940528      6540    4933988   1% /run
/dev/mapper/xubuntu--vg-root  490141656 150647876  314522516  33% /
tmpfs                          24702624      5224   24697400   1% /dev/shm
tmpfs                              5120        24       5096   1% /run/lock
tmpfs                          24702624         0   24702624   0% /run/qemu
tmpfs                           4940524      2564    4937960   1% /run/user/1000
/dev/md124p1                 1922654700 734730036 1090232864  41% /media/mark/2863fe53-a5fa-41d3-90b6-f2cce60c8da1

So I’m still missing something here - any ideas ? It’s almost as if it has added /dev/shm, which it certainly shouldn’t be backing up. But adding /dev/shm to the ignore list makes no difference. Is there any reason why, on Linux, all these ‘not real files’ things should not be ignored by default ?

5

Kopia is not specialised full system backup software but general purpose, cross platform files backup program. It is entirely up to a user to decide what to backup and what not. You want full / backup? You have to learn details of your OS and configure kopia the way you want. Different Linux distros can have different things which should be ignored. And there are many other operating systems kopia can be used with. It would be huge project on its own to build and maintain full system backup exclusion list.

IMO kopia is great if you want to backup your Documents directory but if your requirement is to backup full system you should look for other tools like veeam. Especially if your goal is to have backup solution you can use to restore full system when needed.

6

+1 on that. Additionally - and depending on the change rate of your Linux installation, I’d rather have a backup I can get to independently of any other backup software. E.g. a tar archive, which can be restored with tools already provided by a base OS install. Or a XFS dump, which is still my preferred choice of fs on Linux.