Missing BLOBs on Maintenance - Snapshots appear to work OK

Support
1

Hi all,

New to Kopia but not to backup operations - previous experience in NetBackup, which I know works quite differently to Kopia…

Anyway, I have an issue. Last weekend my S3 provider had a major outage, taking down the bucket my Kopia agents are using. Since the service came back up, snapshots are apparently working, and a quick check of files in the snapshots appear to show no data corruption (although there’s a lot of data to check…). However, quick maintenance operations on both agents are failing complaining about missing blobs. The S3 provider doesn’t think there’s been data loss, but there may have been open file issues as the system was in a “yo-yo state”(sic) before it failed.

Kopia is used as a ‘paranoia’ backup, so missing data isn’t the end of the world as I have a commercial backup solution (crashplan) in use on the (Windows) non-dev machine, so the question really is given the symptoms (I’ll put some logs in below) is it worth trying to recover what I can on this data, with all the risks a repair causes as well, or am I best just wiping the bucket and starting from scratch?

Running quick mainenance:

$ kopia --config-file=/home/alex/.config/kopia/repository-1770980940522.config maintenance run
Running quick maintenance…
Compacting an eligible uncompacted epoch…
Finished quick maintenance.
error compacting single epoch: unable to compact blobs for epoch 42: performance will be affected: CompactEpoch: error adding index to builder: error getting index “xn42_8083a09b7da0695948cd6341047162e5-sda3fe4449af5a05413f-c1”: getContent: BLOB not found

Running full maintenance:

$ kopia --config-file=/home/alex/.config/kopia/repository-1770980940522.config maintenance run --full
Running full maintenance…
GC found 10373 unused contents (677.9 MB)
GC found 2279 unused contents that are too recent to delete (512 MB)
GC found 31263 in-use contents (3.8 GB)
GC found 345 in-use system-contents (296.7 KB)
GC undeleted 0 contents (0 B)
Finished full maintenance.
error rewriting contents in short packs: failed to rewrite 45 contents

Repository status:

$ kopia --config-file=/home/alex/.config/kopia/repository-1770980940522.config repository status
Config file: /home/alex/.config/kopia/repository-1770980940522.config

Description: My Repository
Hostname: amtest-2-26
Username: alex
Read-only: false
Format blob cache: 15m0s

Storage type: s3
Storage capacity: unbounded
Storage config: {
“bucket”: “backups-001168”,
“prefix”: “TestMint”,
“endpoint”: “REDACTED”,
“accessKeyID”: “REDACTED”,
“secretAccessKey”: REDACTED
“sessionToken”: “”,
“roleARN”: “”,
“sessionName”: “”,
“duration”: “0s”,
“roleEndpoint”: “”,
“roleRegion”: “”
}

Unique ID: REDACTED
Hash: BLAKE2B-256-128
Encryption: AES256-GCM-HMAC-SHA256
Splitter: DYNAMIC-4M-BUZHASH
Format version: 2
Content compression: true
Password changes: true
Max pack length: 21 MB
Index Format: v2

Epoch Manager: enabled
Current Epoch: 44

Epoch refresh frequency: 20m0s
Epoch advance on: 20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin: 4h0m0s
Epoch checkpoint every: 7 epochs

Kopia version:

Version v0.22.3 154bf56899228e5c95fb3176b9c6901bbe4ca97b kopia/htmlui 00007b1667f4fb02aee448a54dee2e5c3990b6af built on Tue Nov 25 01:09:40 UTC 2025 runnervmg1sw1

Mainly running through kopia-ui, but using cli commands to get the results in quotable form. This is Linux Mint, but my Windows 11 at home is doing the same thing.y

Thanks in advance for any helpful tips! :slight_smile: