Best method to ensure valid snapshots: snapshot verify vs snapshot fix invalid-files

Support
2

There are many verification methods, depending on what you need.

In the order of lowest- to highest-level:

  1. kopia content verify - will ensure that content manager index structures are correct and that every index entry is backed by an existing file

  2. kopia content verify --download-percent=10 - same as above, but will download 10% of random contents and ensure they can be decrypted properly

  3. kopia snapshot verify - will ensure that directory structures in the repository are consistent by walking all files and directories in snapshots from their roots and performing equivalent of kopia content verify on all contents required to restore each file, but does not download the file

  4. kopia snapshot verify --verify-files-percent 10 - same as #3 but will also download random 10% of all files, this ensures that decryption and decompression is correct.

  5. kopia snapshot fix invalid-files [--verify-files-percent] - performs exactly the same verification as kopia snapshot verify (3&4) for all practical purposes, but it will also write fixed directory entries and manifests.

As of today, it is not recommend to run snapshot fix automatically, only when 3 or 4 detects a problem.

So where does the difference between 1/2 & 3/4 practically matter, you might ask? Imagine a world were some index blobs are deleted and corresponding pack blobs are deleted too. In this case 1/2 will succeed (because index structures are still internally consistent), but if the blobs were needed for some snapshot 3/4 will fail.

It might be worth adding that full maintenance (which happens automatically in both CLI and UI) does #3 every time as part of mark&sweep garbage collection, so technically running verification is optional today as it will be happening regularly anyway. Paying attention maintenance status is not optional in such cases, though and it’s quite hard to observe - this will be improved through notification mechanism in future versions.

Again, all this depends on on the type and value of data, tolerance for risk and cost of data storage but I would generally recommend running something like this daily:

$ kopia snapshot verify --verify-files-percent 1

This will preform full repository scrub every 100 days on average. With tons of data, perhaps decrease that to 0.3 to get full scrub every 300 days. Some folks may want to even perform test restores - again all depends on use case.

11 Likes