Sorry for the noob question, but my Immich backup is 3GB, and I was thinking if using gzip would reduce my disk churn for every backup.
gzip and zstd have gzip --rsyncable option. Will that help so that only changed portions are backed up from a gzipped snapshot?
At present, I am using plain sql dump, without gzip for Kopia backup.
Thanks for your time.
No, it won’t. As the name of the option suggests, this is a feature targeted at rsync specifically and it will force gzip to lower the number of times, it recalculates the best compression mode while compressing an artifact.
Since Kopia is able to compress the data on its own, there’s no need to actually pre-compress the data. Especially when snapshotting database dumps, which can be fed directly into Kopia, so the don’t even make it onto the disk at all.
Can you please help how?
This does not work
sudo docker exec -t immich_db pg_dumpall --clean --if-exists --username=myuser | kopia snapshot create --stdin-filename=PaperlessDB.sql -p “$KOPIA_PASSWORD”
Looks to me, as if you hadn’t connected Kopia to its repo, before trying to perform the snapshot. If you want to connect Kopia at runtime, the repo password alone will not suffice or the user password, if you’re connecting to a kopia server.
What’s the error message, btw.?
Figured out.. just needed to add an empty directory as source
I’m with Budy here, no need to pre-compress, better let Kopia get the unpacked data so it could find repeats in it (possibly) and it can pack lots better than gzip would.
The gzip -rsyncable option is a way to make gzip slightly worse so that it’s buffer size matches the size rsync uses as its smallest blob of comparison and so it resets the byte statistics on each blob, and the hopes are that if you have a large file gzipped with that option and only a few pieces of it changes, then all places where it didn’t change would get the same compressed content, and only those that did change will have to be rsynced. It works somewhat well for certain pieces of data, but it sacrifices a lot of gzip gains since normal gzip would learn what patterns and bytes are most frequent as it goes over the whole file, and if the blob is similar all the way through (like a DB dump which will be a lot of text characters) then it will quickly settle for giving ascii bytes really small codes and non-existant bytes will get the long codes. When you add -rsyncable this “knowledge” is not reused for any later parts of the file, in order to get the “resyncable” property all through the file, so gzip will deliberately “forget” previous statistics. I think the same sort-of applies to parallel gzip where you send of pieces to different cpu cores since they can’t know what the other parts looked like, but you get so much speedup on modern cpus that its a worthwhile tradeoff.
In the end, my suggestion is to let Kopia (which is far more modern than gzip) handle dedupe and compression on its own (possibly with zstd, thats a really good packer) and it will do the best it can.