Snapshot GC failure: Could not undelete referenced content (deleted session blob, BLOB not found) - Kopia 0.22.3 / S3/MinIO

Support
1

Support

Environment:

  • Kopia Version: 0.22.3
  • Storage: S3 (MinIO, endpoint: s3 own storage)
  • Deployment: Docker container (kopia/kopia:latest) via Podman Quadlet on Rocky Linux 9
  • Repository format: v3, Epoch Manager enabled, Current Epoch: 27

Problem:

Since March 14, 2026 every snapshot-gc run fails with:

snapshot GC failure: error running snapshot gc: error iterating contents: 
Could not undelete referenced content: 
{pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e 
1774695556 198443 198471 32 0 f2a5ef14efd7a9d31b8be03cd79d0212 true 3 0}: 
unable to get content data and info: error getting cached content from blob 
"pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e": 
failed to get blob with ID pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e: 
BLOB not found

The content entry has deleted: true – it belongs to an interrupted backup session from March 14. The physical blob no longer exists in S3 (confirmed via blob list | grep). However, Kopia’s GC tries to undelete it and fails because the blob is gone.

What we confirmed:

  • kopia snapshot verify → 356,508 objects, zero errors – all snapshots intact
  • The blob pe15bce50646ecb824423a908daf46592-sb5dddc2d36f62fe913e does not exist in S3
  • full-rewrite-contents has been failing daily since March 5 (growing from 996 → 21,108 failed contents)
  • Index blob count has grown to ~15,000+ due to failed maintenance

What we tried:

  • kopia blob delete on the blob (already gone, exit 0)
  • kopia blob delete on xn/xs/q-epoch blobs referencing the session ID
  • kopia index recover --dangerous-commands=enabled --commit (twice, with server stopped)
  • kopia cache clear (caused SIGSEGV – separate bug?)
  • Clearing cache directory manually + service restart
  • kopia maintenance run --full --safety=none

None of these resolved the issue. After each index recover, the content entry reappears in the next maintenance run.

Question:

Is there a way to permanently remove a deleted: true content entry from the Epoch index when its backing blob no longer exists? The entry seems to survive index recover because it gets re-written from the xn-epoch blobs which still contain it.