Snapshot GC failure: Could not undelete referenced content (deleted session blob, BLOB not found) - Kopia 0.22.3 / S3/MinIO

Support
1

Support

Environment:

  • Kopia Version: 0.22.3
  • Storage: S3 (MinIO, endpoint: s3 own storage)
  • Deployment: Docker container (kopia/kopia:latest) via Podman Quadlet on Rocky Linux 9
  • Repository format: v3, Epoch Manager enabled, Current Epoch: 27

Problem:

Since March 14, 2026 every snapshot-gc run fails with:

snapshot GC failure: error running snapshot gc: error iterating contents: 
Could not undelete referenced content: 
{pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e 
1774695556 198443 198471 32 0 f2a5ef14efd7a9d31b8be03cd79d0212 true 3 0}: 
unable to get content data and info: error getting cached content from blob 
"pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e": 
failed to get blob with ID pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e: 
BLOB not found

The content entry has deleted: true – it belongs to an interrupted backup session from March 14. The physical blob no longer exists in S3 (confirmed via blob list | grep). However, Kopia’s GC tries to undelete it and fails because the blob is gone.

What we confirmed:

  • kopia snapshot verify → 356,508 objects, zero errors – all snapshots intact
  • The blob pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e does not exist in S3
  • full-rewrite-contents has been failing daily since March 5 (growing from 996 → 21,108 failed contents)
  • Index blob count has grown to ~15,000+ due to failed maintenance

What we tried:

  • kopia blob delete on the blob (already gone, exit 0)
  • kopia blob delete on xn/xs/q-epoch blobs referencing the session ID
  • kopia index recover --dangerous-commands=enabled --commit (twice, with server stopped)
  • kopia cache clear (caused SIGSEGV – separate bug?)
  • Clearing cache directory manually + service restart
  • kopia maintenance run --full --safety=none

None of these resolved the issue. After each index recover, the content entry reappears in the next maintenance run.

Question:

Is there a way to permanently remove a deleted: true content entry from the Epoch index when its backing blob no longer exists? The entry seems to survive index recover because it gets re-written from the xn-epoch blobs which still contain it.

2

also having this problem and would appreciate some guidance

3

I have since disconnected, deleted, and rebuilt the S3 storage. This resolved the issue.

4

I’m not on S3 but having this same error and have tried the same things

what exactly do you mean by disconnect, delete and rebuild?

5

I disconnected the S3 repository from the Kopia server, then deleted everything related to Kopia from the S3 storage, reconnected the repository, and had all clients create new snapshots or waited until they did so on their own.

6

Well… by “reconnected” you mean recreate, right? If you deleted everything from the S3 bucket you effectively removed the repo…

7

That’s right. I haven’t found a better solution yet.

8

okay yeah I’m ideally looking for the kind of fix that doesn’t involve re-uploading 5TB of data over a slow-ish connection

9

The problem was that the S3 storage was behind a load balancer that wasn’t configured properly. I configured the S3 storage directly as a repository on the Kopia server, and everything has been working fine ever since. Maintenance runs without errors.