I have KopiaUI set to snapshot every 15 mins. I have noticed that even when I change no files, KopiaUI is uploading 4.6 KB of data every snapshot. What is Kopia uploading if nothing has been changed?
EDIT: Here are an example of the two files KopiaUI is uploading every 15 mins, even though no files are being changed in the snapshot source. The two new files being uploaded every 15 mins are the xn3_* index files and the q* blobs The total file size is always 4.6KB if I do not change any files in my snapshot source.
EDIT2: KopiaUI confirms that no changed files are being uploaded (see Hashed Files 0) but yet 4.6KB of uploaded bytes (of the two files referenced above):
Every time you run kopia snapshot ... a snapshot is actually created. The information of that snapshot is saved in your repo. According to the documentation x files are “indices” and q files are “meta data”. So no “content data” is stored according to your information.
I am guessing this is being done for snapshot retention purposes. Is there no way for Kopia to keep track of empty snapshots without junking up the repo with empty snapshot files? If someone were to run a snapshot every 10 mins, that is 12 unnecessary files being created an hour if there are no files changed in the source.
Backup software must keep somewhere its data. Kopia stores in the destination. Some other software, for example Dupliacti, creates a local database in the source. Having local database causes, imho, more problems than its solves. Possible corruption, various forms of mismatches, problems with rebuild database etc. etc. Kopia solution is more elegant, practical and safer in the long run - you can plugin any time your backup repository and voila…
I do not disagree with this. What I am saying is, there surely must be someway to keep track of empty snapshots without creating junk files in the repo? For example, maybe one file in the repo that keeps track of all retained snapshots. This file can be updated when the snapshot is empty. This would keep the repo clean.
If Kopia always uploads one index and one blob for every snapshot even when no files in the source have been changed, then it a bug that, occasionally, for some snapshots Kopia uploads nothing? Maybe worth getting @jkowalski to chime in here, just in case this is a bug. Sorry if this is intended behavior!
Yeah, all this is to track empty snapshots, which will always create a manifest entry. q blob stores manifest payload JSON and x blob is an index entry.
It should be possible to add a feature (and new policy setting to control it) that avoids tracking unchanged snapshots altogether.
Given current architecture this should actually be quite easy to do. Basically, one should be able to add logic to avoid calling snapshot.SaveSnapshot for identical snapshots. This needs to be done twice - in the CLI mode (command_snapshot_create.go) and in the server mode (source_manager.go).
I’ll be happy to review pull requests for this. This should be a fun small change.
