"BLOB not found" on S3 repository

kopia-0.19.0 on Fedora 41

Storage type: s3

kopia snapshot list
ERROR error listing all snapshot manifests: unable to find snapshot manifests: unable to load manifest contents: error loading manifest content: error getting cached content from blob "q313ce093b876d6af202f38e2e46ad82d-s01149c9e64f7f24b131": failed to get blob with ID q313ce093b876d6af202f38e2e46ad82d-s01149c9e64f7f24b131: BLOB not found

This repo worked a few days ago without issues.

Googled around a bit and tried for example:

kopia index recover --parallel=10 --commit --advanced-commands=enabled

that ran through OK, but the behavior didn’t change.

Any chance to fix that?

I write to that repository from 3 clients.

On 2 of 3 there are snapshots displayed etc / so the repo isn’t completely shot.

On the one client doing the maintenance jobs, I can successfully create new snapshots.

See this:

 kopia maintenance run  --full
Running full maintenance...
Looking for active contents...
ERROR error processing : error verifying kfba63126f82406a7b08a0a30b5d12e62: error getting content info for kfba63126f82406a7b08a0a30b5d12e62: content not found
Finished full maintenance.
ERROR snapshot GC failure: error running snapshot gc: unable to find in-use content ID: error processing snapshot root: error verifying kfba63126f82406a7b08a0a30b5d12e62: error getting content info for kfba63126f82406a7b08a0a30b5d12e62: content not found

pls advise

“funny”: it magically works again.
Leaves me a bit uncertain …

Others might be able to chime in better but you may have caught Kopia during a dedup job run. That could explain it but I’m not exactly well versed in knowing what to look for in the $HOME/.cache/kopia/cli-logs/ of either endpoints.

Would that take over a day? Why doesn’t the software tell me that it does do things, instead of failing with this very frightening “error”?

Kopia doesn’t do any “dedup” runs. Dedup is always performed at snapshot time. Are you maybe referring to a maintenance run. Maintenance runs may include re-packaging of BLOBs. It may be that there was some stale data in the local cache.