I was backing up my video projects with KopiaUI for years until there was a failure and now I’m trying to recover from snapshots and discovering that there’s corruption and some of the files are toast. The recovery is failing as a result. For future reference, if I decide to continue with Kopia, do I need to manually run CLI commands on a schedule to use this software? It’s my mistake for assuming KopiaUI would do everything, and trusting when snapshots completed without showing errors in the UI, but now I’m curious about the recommended workflow.
It would have been great if I could have defined a policy to verify contents on a schedule in KopiaUI. I’ll create another thread to see if I can save my corrupted snapshots.
A full mainteance run basically performs the same tasks as kopia content verify, which checks all the metadata and index files. If your KopiaUI user is the creator (and maintenance owner) of the repository it should run the maintenance tasks automatically. But this does not verify the content (data blobs) of your repository. You need to run kopia content verify --download-percent= yourself to verify the content. This can be done with a script and before/after actions or your systems scheduler.
A configurable maintenance or atleast some information in KopiaUI would really be great. But unless someone comes up with a pull request we gotta do it manually.
All of our repositories are using Kopia Repository Server, and the server runs kopia content verify --download-percent=X% daily. And we are running kopia snap fix invalid-files from time to time but not regularly (yet).
I don’t know why, but kopia snap fix invalid-files keeps finding issues even on stable hardware and connections and when kopia content verify reports no issues. I suspect a bug in the maintenance logic but don’t think anyone will investigate and fix it anytime soon.
Can you elaborate on that? Wouldn’t this bug count as a high-risk bug? After all, it can cause data corruption to remain undetected for everyone who runs no Kopia server.
Well… actually kopia snap fix invalid-files always seems to find something to do… I had it run twice today, the 2nd time also issuing --commit and when I ran it again after some hours, at still found something to do. Soo… not really sure, what that is about, when there were not any new snapshots being made in the meantime.
kopia snap fix invalid-files --commit was last run 6 days ago. 141 snapshots were added to the repository since then. kopia snap fix invalid-files now reports 44 snapshots which need fixing. All of them were made within the last 6 days. kopia content verify --download-percent=100 was run immediately after kopia snap fix invalid-files and reported no errors.
As kopia content verify does not report any errors, I’m pretty confident that my repository is in good shape. But I have no idea what exactly is going on. Thinking about it again I don’t think maintenance is to blame. Maybe kopia content verify and kopia snap fix invalid-files are looking at the data differently.
We run kopia content verify --download-percent daily on our repositories and usually read between 2-10% of the data. Cronjobs or systemd timers) send an e-mail notification when errors are found.
I blamed poor S3 reliability when I first discovered this behaviour but noticed that it happens on local backends aswell. Maybe there’s a bug in kopia snap fix invalid-files that tries to fix snapshots which don’t need fixing? Can’t point my finger at the exact cause.
Will open an issue on Github about it now that other seeing this behaviour as well.
That’s my observation because there are hardly any responses to Github issues or Slack questions and investigating such an issue can be really time consuming.
There is some very valuable information here but is there any decent documentation for the maintenance commands? In particular, what do each of snapshot verify, content verify and maintenance run do?
I would like to add these verification options to my kopia monitoring tool but I’m not sure which one to use, or maybe they should all be used?
Following the discussion in this thread I ran “content verify --verify-files-percent=10” and got quite a lot of errors (>20) so I’m guessing my repository is in a bad shape. However, I don’t understand how it could happen, because I’m backing up to a directly attached internal hard drive, which doesn’t report any errors. I tried running “snapshot verify --verify-files-percent=10” and it didn’t report any errors, so I’m trying again on 100% but either way, since I I don’t know what it actually does, I don’t know what to make of these findings or how to fix them. As a side note it’s also worth mentioning that this verification process is unreasonably slow. It has been running for more than 2 hours now and only processed about 6% of the objects.
kopia content verify does the same as kopia snapshot verify, but doesn’t detect missing blobs.
Running kopia snapshot verify regularly is not necessary, because it is part of a full maintenance run.
It is not recommend to run kopia snapshot fix automatically, only when a problem is detected.
Running kopia snapshot verify --verify-files-percent=X daily is recommended and the best way to make sure your repository is in good shape.
It’s always hard to tell why these errors happen, but I would reommend running an extended SMART test and a memtest.
kopia snapshot fix invalid-files has reliably fixes errors for me in the past. Maybe make a copy of your repository if you don’t feel comfortable using --commit.
You didn’t mention the size of the repository or the hardware you are running on, so it is hard to tell what unreasonably slow means. The command generally is CPU-bound, because it has to decrypt, decompress and hash every single blob. What does your CPU usage look like?
