Can not access repository, failed to get blob

Support
1

I have a large kopia repository in Gdrive. Everything worked fine until yesterday I could not access the repository anymore and got the following error message:

  • Error: error connecting to repository: unable to list sources: unable to find manifest entries: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID qa2328e949a262602138a0c377e1af092-s8209e3abbb2ab48d110: BLOB not found.

I have no clue what caused this for everything was running fine the day before. The only thing changed was upgrading kopia to the new version.

Any idea what to do?

2

Can you try downgrading Kopia to the version that worked and see if the error persists? That is an easy way to rule out if this was caused by a regression. Downgrading is easy, just download the version you had previously. The only thing to note is you cannot downgrade to below v0.9.0 because of the repo change.

Otherwise, are you using the gdrive repo or connecting to gdrive through the rclone repo? When you say you cannot access the repo, do you mean you literally cannot connect to it or that you can connect and you just get errors when trying to run a snapshot or a restore? If the latter, there could be data corruption. Try kopia snapshot verify. One thing that may have happened is a snapshot was running when you upgraded and that causes some issue with blobs that Kopia thinks should be there but are not.

3

Thank you for the quick response.

I reinstalled the previous version v0.11.2 but that did not solve the issue.

I am using the rclone repo.

Normally I use the KopiaUI interface and using this I cannot connect to the rclone repo and it gives the stated error message.
But just now I tried using the Kopia command line commands and that gave the following results:

  1. kopia repository connect from-config --token , connects to the rclone repo without a problem.
  2. kopia blob list; gives no errors.
  3. kopia content list; gives no errors.
  4. kopia index list; gives no errors.
  5. kopia manifest list; ERROR unable to find manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID q886d874d6a8b1df0149ec4dbe8ae24b8-s129a53ef8c78c766110: BLOB not found.

Does this give you any clues?

4

Why this happened, I do not know. Someone else will have to jump in to diagnose.

How to fix it? Try reading

and

to see if anything they recommend fixes the issue for you.

If this were my repo, if all else fails, I would run kopia snapshot fix invalid-files --verify-files-percent=100 --commit. That may or may not work. It worked for

Snapshot GC failure: error running snapshot gc: unable to find in-use content ID.

5

I have tried runnng kopia snapshot fix invalid-files --verify-files-percent=100 --commit. But that did not help. It gave the following result:

C:\Users\Ruud>kopia snapshot fix invalid-files --verify-files-percent=100 --commit
2022-07-13T20:13:54.021704+02:00 Listing blobsā€¦
2022-07-13T21:47:43.194737+02:00 10000 blobsā€¦
2022-07-13T22:56:55.656566+02:00 20000 blobsā€¦
2022-07-13T23:41:19.251981+02:00 Listed 26227 blobs.
2022-07-13T23:41:19.252225+02:00 Listing all snapshotsā€¦
2022-07-13T23:41:47.081074+02:00 ERROR error running maintenance: error running maintenance: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID q9f929af62689a12d7b6405816434c9f5-s3926780f93563acb110: BLOB not found

2022-07-13T23:41:49.365427+02:00 ERROR unable to list snapshot manifests: unable to find snapshot manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID qa2328e949a262602138a0c377e1af092-s8209e3abbb2ab48d110: BLOB not found

Has anybody any ideas what I could try next?

6

Did you read ERROR: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: getCacheForContentID: BLOB not found Ā· Issue #1087 Ā· kopia/kopia Ā· GitHub and try the steps that worked for people there? Specifically, see ERROR: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: getCacheForContentID: BLOB not found Ā· Issue #1087 Ā· kopia/kopia Ā· GitHub and ERROR: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: getCacheForContentID: BLOB not found Ā· Issue #1087 Ā· kopia/kopia Ā· GitHub

7

No sorry, I did glance at this, but it looked rather complicate. Since I did not have any time today, I decided to run the kopia snapshot fix first, because that could run overnight and share the results. Iā€™ll certainly look into this more closely tomorrow.

8

I followed all steps from your link. Found the erroneous index file, but removing this index file from the remote repo did not solve the problem. However I did notice something else which seems odd to me. All remote index files are numbered sequentially from 1_0 thru 1_f, 2_0 thru 2_f, etc. But in my erroneous index file some indexes are missing. The current list is: 5_0, 5_2, 5_3, 5_4, 5_6, 5_7, 5_8, 5_9, 5_a, 5_d, 5_e all dated 11 july 2022. But missing 5_1, 5_5, 5_b, 5_c, and possibly 5_f.
I suppose that this could cause the problems. Is there any way to rebuild the index?

9

Try kopia index recover --parallel=10 --commit. You could also consider including the --delete-indexes command, but I would recommend downloading your indexes to your computer first, in case something goes wrong and you want to put back the ā€˜oldā€™ ones. See index recover | Kopia

Try running a snapshot and maintenance after that. If still errors, you could also try kopia snapshot fix invalid-files --verify-files-percent=100 --parallel=10 --commit again after you try recovering the index.

10

I have run kopia index recover --parallel=10 --commit --delete-indexes. The run ended without errors, but did not solve the problem. The above mentioned index list is exactly the same with the same files ā€œmissingā€. Apparently these files are not missing and the index is correct this way.

Next I ran kopia snapshot fix invalid-files --verify-files-percent=100 --parallel=10 --commit . This ended in error unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID q6a0d05972b5f739d45a46ce481fcda0d-s086049b6da019c69110: BLOB not found
I checked this blob in the rclone repo and q6a_0d0 is empty.
Then I checked all other blobs mentioned in the previous error messages, and all are empty in the rclone repo.

Since I understood from other discussions that kopia will never ever write empty files to the repo, this clearly is in error. But how to solve this?

11

Did you try what is listed at https://kopia.io/docs/advanced/consistency/:

12

Yes I did:

  1. Looking for zero size folders is not a standard function in Gdrive and doing this manually with this number of folders is a nightmareā€¦
    But even then, there appear to be so many empty folders. And I fear that deleting so many folders corrupt things even more.

  2. I tries that as first. But with the same problem.

  3. No strange dates to be found.

  4. I tried this too, but after a while I stashed away so many index files that I might as well delete all of them :wink: So i put them all back again.

13

Someone else will have to jump in if they have any ideas on how to help fix the issues you have. If it cannot be fixed, you may need to wipe the repo and start over, if possible.

FWIW, Im looking at people complaining about corrupt repos, and it seems to be more people that are using rclone rather than a native Kopia repo.

14

I agree that it is better to wipe the repo and start over. But that raises the question how to do that. Can I just delete all content from Gdrive? Wipe the content with kopia content delete? But leaving the root folder intact in Gdrive will result in the same problems I have now?

15

I do not think Kopia has a command to delete a whole repository easily. You could do content delete or snapshot delete or manifest delete, but you would need to do that to each individual respective item.

On the other hand, whenever I need to clear a repo, I literally just go to the platform and delete all the Kopia files in the repo. Presumably, there is a specific folder on Gdrive that contains the Kopia repo. Just log into Gdrive and delete that whole folder. Recreate the folder again in Gdrive. Then reconnect to Gdrive with Kopia. You are literally starting a new repo from Kopiaā€™s perspective. Note that this means you will lose all your policies as well, so you will need to recreate them.

16

Iā€™ve deleted the old repo and created successfully a new one. Recreated the snapshots and policies and itā€™s running at this moment.

Pity that we could not save the old repo but so be it. There is no data loss since the repo just contained backup files which I also have on a Kopia repo on an external drive. It just takes a long time to upload all data anew.

Thank for all your help and advices.

1 Like
17

2 years later. Same error.
ERROR error running maintenance: error running maintenance: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: invalid checksum at qa94279184ddcbea10d3ae0e97fa972d0-s8174fe0142ffcc0212b offset 32 length 543/543: decrypt: unable to decrypt content: cipher: message authentication failed
ERROR could not get defined policy: unable to find policy for source: unable to load manifest contents: error loading manifest content: invalid checksum at qa94279184ddcbea10d3ae0e97fa972d0-s8174fe0142ffcc0212b offset 32 length 543/543: decrypt: unable to decrypt content: cipher: message authentication failed

Iā€™ll report back if I have a discovery but I think this was the cause of this issue.

All was fine until I did a kopia policy set using variables like this.

From one bash script I sourced another which contained the value of default_policy_schedule="0,30 6-23 * * 1-6"

Back to the original script it ran
kopia policy set ${dir} --snapshot-time-crontab='${default_policy_schedule}'

However this threw errors and when echoā€™d to the terminal it revealed:

kopia policy set /root --snapshot-time-crontab='0,30 6-23 client_device_add.sh emp....

Along with the error:
ERROR scheduling policy: invalid scheduling policy: invalid cron expression "0,30 6-23 client_device_add.sh empty ignore_file_patterns.sh kopiastuff.sh

Somehow it ended up inserting the file list of the same directory into that variable :thinking:

From that point forward I started getting those errors:

ERROR error running maintenance: error running maintenance: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: invalid checksum at qa94279184ddcbea10d3ae0e97fa972d0-s8174fe0142ffcc0212b offset 32 length 543/543: decrypt: unable to decrypt content: cipher: message authentication failed

18

I can confirm that that was the cause of this issue. In my case I was working with a new repo so it was easy for me to clear its contents and retest. This proved to be the problem. Solved

2 Likes