I have for a while been trying to get maintenance to run on my repository. I am out of ideas. @jkowalski I would very much appreciate further suggestions.
This is a GCS-backed repository of about 8TB in size which according to the full maintenance pass is over half unused contents.
I have attempted several fix / verify passes. Most recently:
kopia snapshot fix invalid-files --verify-files-percent=100 --parallel=64 --commit
This noted No changes. which I take to mean that every file in every snapshot can be restored. But quick maintenance (and a subsequent full maintenance pass) both failed with:
Running quick maintenance...
Rewriting contents from short packs...
unable to rewrite content "k023fe527f0e542a856357ef34f15eb32": unable to get content data and info: error getting cached content: failed to get blob with ID qa26949ddd76ccf227681a627ca8d1236-s2f5e131e54cda470101: BLOB not found
...snip...
Finished quick maintenance.
ERROR error running maintenance: error running maintenance: error rewriting metadata contents: failed to rewrite 597 contents
The list of missing contents is stable across all my attempts to get maintenance to complete. There are apparently two missing blobs (by name they are index blobs?) responsible for the missing 597 contents.
q8fcccb93937da98dcad8d0999169fbb4-sfac90825bf56232b101
qa26949ddd76ccf227681a627ca8d1236-s2f5e131e54cda470101
Perhaps these two blobs aren’t actually needed?
kopia content verify --parallel=64
...snip...
Verified 5725414 contents, 0 errors, estimating...
Finished verifying 5751128 contents, found 0 errors.
kopia content verify --include-deleted --parallel=64
...snip...
ERROR error content xfec955d5902c8e72c44627487dfc94b1 depends on missing blob qa26949ddd76ccf227681a627ca8d1236-s2f5e131e54cda470101
Finished verifying 16492595 contents, found 597 errors.
ERROR encountered 597 errors
But that observation doesn’t help me figure out how to get the maintenance pass to complete successfully.