Dealing with snapshot corruption

Support
1

Hi,

first off, thanks for the awesome work you guys are doing! Kopia really is the best thought through solution I could find. After finally deploying, I have a crucial question, though:
How do I deal with errors reported by kopia snapshot verify?
I’m seeing some errors of the sort

2021-11-24T08:45:44.004217Z getContentInfoReadLocked(693effc76b727cb7aef071cac74b30b1) error invalid checksum at p676b57f40ef4d015681c654ebd9a36e3-sffed7182750f806d10a offset 8578957 length 2746658/2746658: decrypt: unable to decrypt content: cipher: message authentication failed

in the log.
In itself, these warnings aren’t alarming for me, because I accidentally tried what it could be like to back up over an unstable connection to a device that might experience a power outage :laughing:.

My question is - and sorry, I could not find any documentation on this and want to be sure - how do I proceed from here? Is it

  • enough to run create snapshot with the --force-hash option frequently to discover and fix these kinds of corruptions?

  • necessary to run any other command, to tell kopia to get rid of these unreadable chunks and try to find them in the source and reupload them?

  • completely wrong what I assume kopia works like?

Also, and on a sidenote, I’m using version 0.9.5 and have set a policy specifying that all content (no matter size or extension) should be zstd compressed. That is also what policy get reports. However, looking at the stats, I find that only a small fractiion (400 MB / 140 GB) have been compressed at all. Most of the files do not have extensions, but according to the policy this should not matter. I setup a test repository and backed up “.dat” and “.txt” files with the same settings and only the txt files were compressed. Did i miss something?

Thank you very much for your support!

2

Currently there’s not a great way to recover from such corruption, other than migrating to a new repository.

You need to create second repository and use kopia snapshot migrate. To make sure you’re not mixing those up, always pass --config-file:

kopia --config-file /path/to/new.config repository create ...
kopia --config-file /path/to/new.config snapshot migrate --source-config /path/to/old.config
kopia --config-file /path/to/old.config repository disconnect

This will copy snapshot from a corrupted repository to a clean ones - any invalid files due to unreadable chunks will be treated the same way as if there were bad sectors/read errors on the original disk - they will be reported as errors and ignored, so you will end up with a clean repository that’s missing some files.

1 Like
3

I see, thank you very much for the quick reply! I will try and see if this procedure is feasible for my purposes.
If I may suggest a possible future feature for consideration in this context: It would be great if

  • the snapshot function could either read an output of verify and reupload erroneous files or
  • verify could delete invalid files, so the next snapshot would reupload them and / or
  • snapshot would have a --force-reupload [size or percent limit] option so that over time errors from previous backups would be resolved.

I don’t know if any of these options can agree with the architecure of kopia, but I imagine an “on-the-fly” solution like that would be highly beneficial for many appliances and also increase effective robustness of this backup solution.