I am using Rclone with a Google Drive (Application Folder) backend. I have recently been getting errors during full maintenance. When I look in the Application Folder for Kopia, those blobs are completely missing. The command kopia snapshot fix invalid-files --commit does not solve the issue. I do not receive errors on snapshot completion. Also of note, for a few weeks before the 0.12.1 update, the KopiaUI program would open with a blank screen even after a restart.
Error: error rewriting contents in short packs: failed to rewrite 1994 contents.
09:29:16.414 Rewriting content kcd470ac5a23fb1a5a9801d54c25e97e0 (1035 bytes) from pack qc99b1da224bcb7d58d0bec95a6e5283b-s7ffbc6ca0472346f10f (deleted) 889h38m41.414864837s
09:29:16.415 unable to rewrite content "x5a43ca88291c0d07270b097d3b847489": unable to get content data and info: error getting cached content: failed to get blob with ID q9076c3bba061ef4b448301eee43fb8f9-sfef153ba1076e0e910f: BLOB not found
09:29:16.415 Rewriting content ke72421e6f37082e98b11105866d26f77 (15881 bytes) from pack qc99b1da224bcb7d58d0bec95a6e5283b-s7ffbc6ca0472346f10f (deleted) 889h38m40.415358649s
09:29:16.415 unable to rewrite content "xb8cc6167d5fb535b80a8d3a501b0bad3": unable to get content data and info: error getting cached content: failed to get blob with ID q9076c3bba061ef4b448301eee43fb8f9-sfef153ba1076e0e910f: BLOB not found
09:29:16.415 Rewriting content kee1108fbd11e65da4f7e0c56e4cc1040 (760 bytes) from pack qc99b1da224bcb7d58d0bec95a6e5283b-s7ffbc6ca0472346f10f (deleted) 889h38m40.415854943s
09:29:16.416 unable to rewrite content "xee82f48920e49893721eb76bad0aa835": unable to get content data and info: error getting cached content: failed to get blob with ID q9076c3bba061ef4b448301eee43fb8f9-sfef153ba1076e0e910f: BLOB not found
09:29:16.416 Rewriting content x6132e8898eca3d51c09b4f631503aca6 (186 bytes) from pack qc99b1da224bcb7d58d0bec95a6e5283b-s7ffbc6ca0472346f10f (deleted) 889h38m30.416327592s
09:29:16.955 unable to rewrite content "x6132e8898eca3d51c09b4f631503aca6": unable to get content data and info: error getting cached content: failed to get blob with ID qc99b1da224bcb7d58d0bec95a6e5283b-s7ffbc6ca0472346f10f: BLOB not found
Seems like the files don’t even exist. Can you manually check your Gdrive to see if the files actually exist? Or mount the path manually using Rclone and check if you can see the files.
If everything is in order and all else fails, disconnect the repository, clear the kopia cache and reconnect again.
@stpr I checked and the files do not exist. For a few of the blobs the directories they should be in are empty, and for another few the directory doesn’t even exist.
That’s a problem… perhaps disconnect the repository, clear Kopia cache, reconnect, and try again? If this is an issue with inconsistent cache, then this should fix it. However, if not, then there is a bigger issue at hand.
After I reconnected I received a different full maintenance error. I ran kopia snapshot fix invalid-files --commit and fixed 80 snapshots. Most of the errors were like this:
WARN removing invalid file heimdall/www/icons/sickchill.png/sickchill.png due to: verify object: error getting content info for acf969da2d5d641831b0fe5a7ead8e7c: content not found
WARN removing invalid file heimdall/www/icons/slack.png/slack.png due to: verify object: error getting content info for b5112838aa0bc2473d5bf4224651f6fc: content not found
WARN removing invalid file heimdall/www/icons/streama.png/streama.png due to: verify object: error getting content info for f00f5ce4e950fe2d1eb8642c7112634f: content not found
WARN removing invalid file heimdall/www/icons/statping.png/statping.png due to: verify object: error getting content info for 7490b9e4421060e3416eb663e73c5e11: content not found
WARN removing invalid file heimdall/www/icons/stash.png/stash.png due to: verify object: error getting content info for 8c894a59643322afcc2c0495658e6fdf: content not found
WARN removing invalid file heimdall/www/icons/synology.png/synology.png due to: verify object: error getting content info for 976ce856bacd03a32869ebc1c452a816: content not found
WARN removing invalid file heimdall/www/icons/synclounge.png/synclounge.png due to: verify object: error getting content info for 55fe822b7b0843dc9d5cd3735f1d56c6: content not found
WARN removing invalid file heimdall/www/icons/thelounge.png/thelounge.png due to: verify object: error getting content info for 52649ea85d3c8d37ad509b49a543fb08: content not found
WARN removing invalid file heimdall/www/icons/tasmoadmin.png/tasmoadmin.png due to: verify object: error getting content info for add5c4a6682ca7bf3f7fb863eaaf2c88: content not found
WARN removing invalid file heimdall/www/icons/trilium.png/trilium.png due to: verify object: error getting content info for 71b45099df03932769368c7c19e49613: content not found
WARN removing invalid file heimdall/www/icons/typo3.svg/typo3.svg due to: verify object: error getting content info for 296201b0b0a29b5b758676a67e161ef1: content not found
WARN removing invalid file heimdall/www/icons/traefik.png/traefik.png due to: verify object: error getting content info for 051875aeb66b1641f4bd193a71fad523: content not found
WARN removing invalid file heimdall/www/icons/truenas.png/truenas.png due to: verify object: error getting content info for 703da364de538185cc8d952b58b7f13b: content not found
Once the snapshot repair was complete, I noticed that it had deltas of -157GB. Basically lost all the files that were part of the snapshots.
Browsing the snapshots in the GUI shows many files with an .INVALID. prefix and incorrect file sizes.
Sounds a bit alarming that you have all these errors, and you seem to indicate that you didn’t see any errors during the snapshot creation. Something is amiss.
If you have the opportunity to do so, perhaps it might be a good idea to create a fresh repo and see if the files are being uploaded correctly.
The new snapshots worked. Previous data is still missing I also have updated rclone as well. I have another instance of this issue on a different g-suite setup at home. I am wondering if I should create a github issue since this is pertaining to data-loss in the archive.
You should indeed file an issue, with as much info as I can. Any data loss that is due to Kopia should be tracked down and fixed.
Out of curiosity, what OS is this on? You mentioned updating rclone, was it terribly out-of-date? Anything that could help narrow down the culprit will be helpful for other users who may end up in the same situation as yours.
I am a bit surprised though since GDrive is not terribly uncommon, and someone should have reported issues with it. You seem to suggest that you reproduced the same issue on a different system and account, so it is probably worth reporting it on GitHub.
I’m a little late here. There’s a scattering of reports throughout this forum of similar issues. I also had something similar happen to me.
From reading documentation and hanging around the forums, kopia has selected some data to be removed from the repository during maintenance. It could be aged out, administratively removed, etc. Kopia is functioning ok.
Kopia’s next job is to try and consolidate content remaining in blob files and rewriting that content into a different blob along with other content from other blob files. At this point, the missing content is supposed to be in the repository, but there are two scenarios here. One possibility is that the content was recently removed. The would represent actual data loss in the sense that it is missing from the repository before scheduled age-out; I’m not sure we can recover from this situation, but maybe it doesn’t matter because the user removed the data on their own and will never ask to recover it from the administrator. In my situation I couldn’t figure out a way to query kopia to determine what content was supposed to be in a particular blob. It would have helped to know if I should be more concerned.
The other option is that the data still remains on the host itself. Since the blob was no longer valid, I did a "kopia blob delete <blobs>". It doesn’t “fix” anything because the content is still missing but now kopia won’t try looking for content in that blob.
Kopia snapshot create seems to look a file’s metadata to determine if it should upload data to the repository. In my situation the data on disk hasn’t changed so kopia didn’t do anything and my repository was still missing content. To force kopia to rescan everything I did a “kopia create --force-hash=100 <stuff to snapshot>”
Now kopia will compare what is on disk with what is in the repository. It will figure out that some data is missing and upload that missing data. It will also rehash that and then update all other content indexes so that data is no longer missing.
This all worked for me, but your situation may be different. I am not a kopia developer
