# Is one bit flip enough to crash a Kopia repo? - How to recover from bit rot? **URL:** \ **Category:** Support\ **Created:** [March 14, 2021, 4:21pm UTC](https://kopia.discourse.group/t/is-one-bit-flip-enough-to-crash-a-kopia-repo-how-to-recover-from-bit-rot/316 "2021-03-14T16:21:40Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![horo](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/horo/32/118_2.png) [@horo](https://kopia.discourse.group/u/horo)\ **Post date:** [March 14, 2021, 4:21pm UTC](https://kopia.discourse.group/t/is-one-bit-flip-enough-to-crash-a-kopia-repo-how-to-recover-from-bit-rot/316/1 "2021-03-14T16:21:41Z")
A friend and I are trying out a bunch of backup tools (namely Duplicacy, Restic and Kopia) to find out which one meets our needs best (for personal use). One test is to simulate the catastrophic scenario of [bit rot / data rot / data degradation](https://en.wikipedia.org/wiki/Data_degradation) by corrupting the repository by flipping a bit in the data sections of one of the blob files. _On the positive side_, Kopia performed really good, is really easy to use and is able to recover all other files, that are not affected by the corruption. Unfortunately we are not able to “repair” the repo into a state where at least the new snapshots do not rely on the corrupted blob but create a new blob from still existing source data. **Is it possible to tell Kopia to detect and rewrite / replace corrupted or deleted blobs, when the source files are still available?** Besides `repository repair` we also tried to delete the blob and the associated content object using `blob delete` and `content remove` commands. We then deleted cache and created another snapshot in the hope, that it will create a new blob for the picture that was corrupted in the backup. Unfortunately this newly created backup still refers to the corrupted/deleted blob and fails to restore the image as well. Thanks in advance 🙂 ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [March 14, 2021, 4:31pm UTC](https://kopia.discourse.group/t/is-one-bit-flip-enough-to-crash-a-kopia-repo-how-to-recover-from-bit-rot/316/2 "2021-03-14T16:31:41Z")
There are some missing features here, so it’s not optimal UX yet, but today you can remove the damaged content from the index (using `kopia content rm `). This will actually corrupt the repository a bit more temporarily, but you should be able to create a snapshot of the file with `--force-hash=100` to reupload the original data which will fix it. This is because kopia uses content-addressable storage and same data will result in same content IDs (within the same repository). There’s probably more streamlined experience here that could be built. Longer term we’ll probably have some redundancy codes to recover from bitrot automatically. In the short term, using managed cloud storage or a filesystem which handles bitrot like ZFS/BTRFS and performing regular scrubs is recommended. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [March 14, 2021, 4:32pm UTC](https://kopia.discourse.group/t/is-one-bit-flip-enough-to-crash-a-kopia-repo-how-to-recover-from-bit-rot/316/3 "2021-03-14T16:32:27Z")
(I think the --force-hash=100 was the critical bit here, without it Kopia will simply assume that content already exists based on modification times of the file). ---
**Author:** ![horo](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/horo/32/118_2.png) [@horo](https://kopia.discourse.group/u/horo)\ **Post date:** [March 14, 2021, 6:50pm UTC](https://kopia.discourse.group/t/is-one-bit-flip-enough-to-crash-a-kopia-repo-how-to-recover-from-bit-rot/316/4 "2021-03-14T18:50:17Z")
Thank you for the quick reply! That worked great! In case someone else comes along here, this is what we did in detail: ``` kopia snapshot verify --all-sources --verify-files-percent=100 ``` gave us the following output: ``` Found 4 objects, verifying 1, completed 0 objects. failed on root@osboxes:/home/osboxes/Desktop/source@2021-03-14 13:37:01 EDT/horoo.jpg: error reading object Z385d923526dea28c53a3a23ad9179eb1: unexpected content error: invalid checksum at pb914b2361932491cb17acf7f82d22050 offset 754 length 107799: decrypt: cipher: message authentication failed kopia: error: encountered 1 errors, try --help ``` which revealed the content ID: `Z385d923526dea28c53a3a23ad9179eb1`. We then needed to remove the `Z` at the beginning and use it to remove the content object: ``` KOPIA_ADVANCED_COMMANDS=enabled kopia content rm 385d923526dea28c53a3a23ad9179eb1 ``` After that, we created a new snapshot using the `--force-hash=100` option: ``` kopia snapshot create /path/to/source/ --force-hash=100 ``` This recreated the corrupted file and even the first backup could be restored again. 😄 👍 ---
**Author:** ![horo](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/horo/32/118_2.png) [@horo](https://kopia.discourse.group/u/horo)\ **Post date:** [March 14, 2021, 6:50pm UTC](https://kopia.discourse.group/t/is-one-bit-flip-enough-to-crash-a-kopia-repo-how-to-recover-from-bit-rot/316/5 "2021-03-14T18:50:49Z")
Now there’s another issue: In case the original file has been deleted, it is not possible to repair the repo using the steps above. The only thing that we could think of to get rid of the error messages was deleting all snapshots, that refer to the corrupted data. As `snapshot verify` only reveals the date of the latest snapshot that is affected, one would have to delete and verify over and over again until all broken snapshots are removed. For a large repo this would be extreeemly time consuming. Is it possible to list all snapshots that use a specific content object? Or is it possible to somehow “hide” the error for future invocations of `snapshot verify`? ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [March 14, 2021, 9:56pm UTC](https://kopia.discourse.group/t/is-one-bit-flip-enough-to-crash-a-kopia-repo-how-to-recover-from-bit-rot/316/6 "2021-03-14T21:56:22Z")
That’s great to hear. We have this idea to be able to remove references to some objects permanently, but it’s not implemented yet: > > > There are some scenarios where you may want to get rid of objects (and their references) after they have been snapshotted... PRs as usual are welcome.