Is there a way to prevent maintenance and snapshots from running simultaneously. I can’t control the timing of either and it seems if one is running, the other should wait until it’s completed.
Like a full maintenance job started 48 minutes ago in my repo and then a snapshot started itself 45 minutes ago and now kopia is running at like 300% cpu. Is there some way to prevent this collision so if maintenance is running, snapshot waits until completion and vice versa? Thanks.
There is no such functionality built into kopia today. You have to switch to manual:) Then you can do all as you like.
Turn automatic maintenance off:
# quick maintenance
$ kopia maintenance set --enable-quick false
# full maintenance
$ kopia maintenance set --enable-full false
Schedule or run manually whenever it suits you:
# quick maintenance
$ kopia maintenance run
# full maintenance
$ kopia maintenance run --full
More details in docs.
I would probably leave quick maintenance on auto and only schedule full at the time when I do not run snapshots.
I’m aware of how to turn maintenance on and off and how to do it manually or automatically. That’s not my question.My question is about preventing snapshots and maintenance from running at the same time.
As I said - you have to manage it yourself if it is what you need.
Actually it is exactly what I am doing. Once a night (enough for me) I run my own kopia shell script where I run snapshot(s) and maintenance in sequence instead of together.
Right but I like my snapshots running hourly and full maintenance every 24h and obviously that creates overlap. Maybe this is something for the developers to think about - I can put it as a feature request in the git - because it seems to me that the management of the snapshot and maintenance schedule and coordinating between them should be happening within Kopia itself rather than the end-user juggling them.
Afaik, snapshots and maintenance do not interfere with one another. The only times I am halting the automatic maintenance is when I perform a repo sync-to-S3.
When they run simultaneously as opposed to sequentially, CPU usage is through the roof.
Well… do you really need to run the full-maintenance daily? What kind of workload requires that? To be of any use, you’d need to have fragments/files, which have only a very limited lifespan - otherwise asking Kopia to remove and re-organize its blobs wouldn’t make much sense.
And 300% CPU sounds not bad. Thats 3 cores at 100% - unless you have a very ancient CPU, which only has 4 cores, I wouldn’t object about the CPU load, but rather the I/O load, which is what usually slows systems down.
My “need” I guess is to have any corruption identified and corrected. One of the reasons I switched to Kopia is I’ve used Veeam (image-based backup) which had, at the time, no maintenance equivalent and the result was occasional corrupted unusable backups which pretty much made the whole concept of a backup useless. What I like about Kopia is ongoing maintenance - gives me peace of mind.
Kopia is backing up an SSD so I’m not sure I/O would really be an issue with that kind of drive?
My CPU is relatively ancient - it’s only 2 cores so the CPU usage matters to me more than the I/O and, as the backup is going over my LAN which is gigabit, networking isn’t a really an issue either.
Very well, but that is not what maintenance does. What you want is probably a run off
kopia snapshot verify --verify-files-percent=100 <Snapshot ID>
Please be aware, that this command will download and check all data from that snapshot and you will need to get the latest snaphot id for it to run.
You can also get a general view of your repo by issueing a
kopia content verify
That will check all your repo’s contents and will output an error, if anything fishy turned up. Its also quite fast, since it only checks if all content files are available. If you want to spot bit rot on your drive, you will have to go with snapshot verify and then with 100% as well. Usually snapshot verify only targets a fraction of the repo’s data. However, both commands can be run in the off hours…
Thanks for that info. I’m ultimately going to move my Kopia repo to BTRFS or ZFS drives to avoid bitrot. You’re right then I’m probably over-maintenancing and in reality Kopia, being a file-based backup rather than an image-based backup, avoids the problem Veeam had where corruption in a single incremental image destroyed access to the entire backup chain. With Kopia it seems the consequences of corruption would be much more limited.
Sure - however, BTRFS and ZFS (although I am loving ZFS) will introduce some overhead, since they’re both COW (Copy ON Write) filesystems. Should you choose to go with ZFS, make sure to adjust the sharding of the repo, otherwise your perfadjusormance will tank… I’ve seen such also with BTRFS and I’d never use BTRFS as a base for a Kopia repo - you may well get past that with evaluating the proper adjustments, but I chose to not venture into that.
I am running my repos on XFS filesystems and actually couldn’t be happier with the performance. Also, even in ZFS you’d have to schedule a scrub manually or automate it via cron… and you don’t want to do that daily either…
Yeah CoW systems are… painful. I used REFS on Windows back in the day and like many people permanently lost data (shudder) and recently I’ve been looking at CEPH/Rook. Sadly, none of it is as easy as NTFS which, for better or worse, is readable everywhere unlike the Linux-based systems which Windows refuses to support despite having WSL built in.