Snapshot GC failure: Could not undelete referenced content (deleted session blob, BLOB not found) - Kopia 0.22.3 / S3/MinIO

Support
1

Support

Environment:

  • Kopia Version: 0.22.3
  • Storage: S3 (MinIO, endpoint: s3 own storage)
  • Deployment: Docker container (kopia/kopia:latest) via Podman Quadlet on Rocky Linux 9
  • Repository format: v3, Epoch Manager enabled, Current Epoch: 27

Problem:

Since March 14, 2026 every snapshot-gc run fails with:

snapshot GC failure: error running snapshot gc: error iterating contents: 
Could not undelete referenced content: 
{pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e 
1774695556 198443 198471 32 0 f2a5ef14efd7a9d31b8be03cd79d0212 true 3 0}: 
unable to get content data and info: error getting cached content from blob 
"pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e": 
failed to get blob with ID pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e: 
BLOB not found

The content entry has deleted: true – it belongs to an interrupted backup session from March 14. The physical blob no longer exists in S3 (confirmed via blob list | grep). However, Kopia’s GC tries to undelete it and fails because the blob is gone.

What we confirmed:

  • kopia snapshot verify → 356,508 objects, zero errors – all snapshots intact
  • The blob pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e does not exist in S3
  • full-rewrite-contents has been failing daily since March 5 (growing from 996 → 21,108 failed contents)
  • Index blob count has grown to ~15,000+ due to failed maintenance

What we tried:

  • kopia blob delete on the blob (already gone, exit 0)
  • kopia blob delete on xn/xs/q-epoch blobs referencing the session ID
  • kopia index recover --dangerous-commands=enabled --commit (twice, with server stopped)
  • kopia cache clear (caused SIGSEGV – separate bug?)
  • Clearing cache directory manually + service restart
  • kopia maintenance run --full --safety=none

None of these resolved the issue. After each index recover, the content entry reappears in the next maintenance run.

Question:

Is there a way to permanently remove a deleted: true content entry from the Epoch index when its backing blob no longer exists? The entry seems to survive index recover because it gets re-written from the xn-epoch blobs which still contain it.

2

also having this problem and would appreciate some guidance

3

I have since disconnected, deleted, and rebuilt the S3 storage. This resolved the issue.

4

I’m not on S3 but having this same error and have tried the same things

what exactly do you mean by disconnect, delete and rebuild?

5

I disconnected the S3 repository from the Kopia server, then deleted everything related to Kopia from the S3 storage, reconnected the repository, and had all clients create new snapshots or waited until they did so on their own.

6

Well… by “reconnected” you mean recreate, right? If you deleted everything from the S3 bucket you effectively removed the repo…

7

That’s right. I haven’t found a better solution yet.

8

okay yeah I’m ideally looking for the kind of fix that doesn’t involve re-uploading 5TB of data over a slow-ish connection

9

The problem was that the S3 storage was behind a load balancer that wasn’t configured properly. I configured the S3 storage directly as a repository on the Kopia server, and everything has been working fine ever since. Maintenance runs without errors.

10

I fixed this by creating new snapshots, each one being a sub directory of the main snapshot content (there were about 8 sub directories so I created that many new snapshots)

because the data for these snapshots already existed in the repository, they did not require a full upload and creating them was quite quick

once they were all created, I was able to delete the original snapshot entirely. maintenance has been working ever since

so not the simplest or most practical solution in all cases but it worked for me, and maybe posting it here will help someone else as well

11

I’m in a similar situation, full maintenance is failing (maybe it’s been failing since 2022? I just noticed this :laughing:) and I’d prefer not to start a fresh S3 repo because my Canadian internet connection would take over a week to re-upload.

In my case, I have all the data locally, so I would be comfortable running a dangerous command like “nuke the problematic bits and rebuild from my local files” if such a thing exists.

12

That’s a good idea. I did something similar and created a new snapshots of the subdirectories that contained the bulk of the size. If you know where the problematic blob is, maybe you could just create a snapshot of that directory?

Anyway, hashing those terabytes took some time but it only uploaded a few KB. Then i was able to run full maintenance, and delete the new snapshots I made. Now the old snapshots are working again.

Leaves me with a question: what is undeleting in the context of Kopia? Why does Kopia need to undelete anything?