Repairing "missing blob" errors

Support
1

Before I do something egregiously stupid, I’d appreciate someone else confirming or correcting my thinking here.

I had been remiss in running kopia repository maintenance --full and when I got around to it:

Processed 30864 contents, discovered 161218…
Finished full maintenance.
ERROR: snapshot GC failure: error running snapshot gc: unable to find in-use content ID: error walking snapshot tree: error reading directory: unable to open object: k25a0a66a8f229cab93a7eb95aff0e179: unexpected content error: getCacheForContentID: BLOB not found

My presumption here is that at some point, object k25a0a66a8f229cab93a7eb95aff0e179 never got written to the repository.

Is there any way to “fix” this other than removing all snapshots that contain k25a0a66a8f229cab93a7eb95aff0e179?

Ideally, I would discover that erroring snapshots would be the only ones containing this directory ref and I could just delete those? Otherwise, I expect that I will have to partially restore the damaged snapshots?

2

What’s your backend provider? There’s currently no universal answer to this type of question - it all depends on what exactly happened. Ping me on slack so we can investigate.

3

I’m using GCS. I’ll ping you. I kicked off a script to validate all of the snapshots under the presumption that it would at least tell me which of my snapshots are good.

4

Any progress here?
I have the same problem with “BLOB not found”…

5

I have finally gotten back to really digging into this. This is with a GCS backend.

TL;DR: I did a sequence of verify / maintenance cycles and I’m stilling getting a profusion of “missing blob” errors.

More details:

kopia snapshot fix invalid-files --commit
# ...time and logs noting that some file were fixed
kopia maintenance run --full
# ...time...
# final errors dumping errors featuring "missing blob" per above

I searched here a bit and saw some posts from jkowalski that suggested that I could instead do (and so did) the following:

kopia snapshot verify --verify-files-percent 100
# ...time... and no additional corrections to the backend
kopia maintenance run --full
# ...time... and still "missing blob" errors

So any suggestions?

I am imagining trying to use kopia repository sync-to gcs to a new repository? Presumably if the verify passed then a copy of only the valid contents would be error free?

6

Perhaps I mis-used the snapshot verify command? I will run it again like so:

kopia snapshot verify --verify-files-percent=100