Please - help me understand - why maintenance failing - why too many index blobs
Been on this all week. Can’t solve it. Outside of wiping out backups on multiple servers and starting anew I’m out of options. There must be a solution.
I have kopia installed on about 6 servers. All servers are running Ubuntu 22.04 Server. All original data lives on ZFS Raid-Z2 though the backup drive is native ext4 and not ZFS.
We’ll use one example where when I run this command to get the number of index blobs:
NUMBER OF BLOBS:
sudo kopia --config-file=$configs -p $password index list | wc -l
Which returns 92602.
Given it is recommended to be less than 1000 index blobs this is obviously a real problem.
We have other servers (though not all) experiencing similar very high index blobs as well.
It is clear to me at some point daily automated maintenance stopped working so the blobs continue growing.
This command shows returns:
sudo kopia --config-file=$configs -p $password index list | wc -l
92602
I am unclear on exactly 2 things - how to:
- methodically get index blobs back under the recommended 1000
- most importantly - identifying the reason this condition came about so it can be avoided
MAINTENANCE STATUS:
Since running maintenance is the prudent thing to do to keep the backup system running smoothly ongoing, and knowing this many blobs presents a problem for maintenance I move on to geet some status on maintenance by running:
sudo kopia maintenance status --config-file=$configs -p $password | grep -E "ERROR|advance-epoch"
Reveals - ERROR: error compacting single epoch: unable to compact blobs
Full reply for kopia maintenance status below:
kopia maintenance status --config-file=$configs -p $password | grep -E "ERROR|advance-epoch" -A4
advance-epoch:
2024-11-14 22:16:45 PST (0s) SUCCESS
2024-11-12 22:19:07 PST (0s) SUCCESS
2024-11-11 22:11:05 PST (0s) SUCCESS
2024-11-10 22:14:28 PST (0s) SUCCESS
--
2024-11-15 22:21:45 PST (0s) ERROR: error compacting single epoch: unable to compact blobs for epoch 62: performance will be affected: CompactEpoch: error adding index to builder: error getting index "xn62_f695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e-c1": decrypt blob: error decrypting BLOB xn62_f695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e-c1: ciphertext too short: 0
2024-11-14 22:16:44 PST (0s) SUCCESS
2024-11-12 22:19:07 PST (0s) SUCCESS
2024-11-11 22:11:04 PST (1s) SUCCESS
2024-11-10 22:14:28 PST (0s) SUCCESS
--
2025-12-06 04:48:08 PST (7m39s) ERROR: failed to rewrite 1 contents
2025-12-04 21:13:01 PST (16m53s) ERROR: failed to rewrite 1 contents
2025-12-02 10:47:32 PST (8m26s) ERROR: failed to rewrite 1 contents
2025-11-30 05:31:13 PST (5m42s) ERROR: failed to rewrite 1 contents
2025-11-28 23:46:50 PST (12m36s) ERROR: failed to rewrite 1 contents
generate-epoch-range-index:
2024-11-14 22:16:45 PST (0s) SUCCESS
2024-11-12 22:19:07 PST (0s) SUCCESS
2024-11-11 22:11:05 PST (0s) SUCCESS
So clearly this blob “xn62_f695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e-c1” is a problem child
MANUAL CHECK OF BLOB xn6/2_f/695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e-c1.f reveals
ls -l /MYBACKUPS/DATA/xn6/2_f/695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e-c1.f/
Returns:
/ grafana-server.service.d/ smbd.service snap-core24-1225.mount sockets.target.wants/apache2.service graphical.target.wants/ sm_dhcp_monitor.service snapd.mounts.target.wants/ sshd-keygen@.service.d/bluetooth.target.wants/ hibernate.target sm_kopia_webui@.service snap-firefox-7355.mount sshd.servicechronyd.service hybrid-sleep.target sm_on_network_alive.service snap-firefox-7423.mount sudo.servicecloud-final.service.wants/ iscsi.service sm_tap0_br0.service snap-gnome\x2d42\x2d2204-202.mount suspend.targetcloud-init.target.wants/ libvirtd.service.d/ sm_vitals_genstats.service snap-gnome\x2d42\x2d2204-226.mount sysinit.target.wants/CompuMatterRemoteSupportAgent.service mdmonitor.service.wants/ sm_vitals_genstats.timer snap-gnome\x2d46\x2d2404-125.mount syslog.servicedbus-fi.w1.wpa_supplicant1.service multipath-tools.service snap-bare-5.mount snap-gnome\x2d46\x2d2404-145.mount systemd-networkd.servicedbus-org.bluez.service multi-user.target.wants/ snap-certbot-5057.mount snap-gtk\x2dcommon\x2dthemes-1535.mount systemd-resolved.servicedbus-org.freedesktop.Avahi.service netplan-ovs-cleanup.service snap-certbot-5214.mount snap-lxd-36558.mount systemd-timesyncd.servicedbus-org.freedesktop.ModemManager1.service network-online.target.wants/ snap.certbot.renew.service snap-lxd-36918.mount timers.target.wants/dbus-org.freedesktop.nm-dispatcher.service nmbd.service snap.certbot.renew.timer snap.lxd.activate.service var-snap-firefox-common-host\x2dhunspell.mountdbus-org.freedesktop.resolve1.service oem-config.service.wants/ snap-certbot\x2ddns\x2dcloudflare-4697.mount snap.lxd.daemon.service vmtoolsd.servicedbus-org.freedesktop.thermald.service open-vm-tools.service.requires/ snap-certbot\x2ddns\x2dcloudflare-4856.mount snap.lxd.daemon.unix.socket winbind.servicedbus-org.freedesktop.timesync1.service paths.target.wants/ snap-core-17212.mount snap.lxd.user-daemon.service zed.servicedefault.target.wants/ printer.target.wants/ snap-core-17247.mount snap.lxd.user-daemon.unix.socket zfs-import.target.wants/display-manager.service redis.service snap-core20-2669.mount snap.mesa-2404.component-monitor.service zfs-mount.service.wants/display-manager.service.wants/ rescue.target.wants/ snap-core20-2682.mount snap-mesa\x2d2404-1165.mount zfs.target.wants/emergency.target.wants/ sleep.target snap-core22-2139.mount snap-mesa\x2d2404-912.mount zfs-volumes.target.wants/final.target.wants/ sleep.target.wants/ snap-core22-2163.mount snap-snapd-25202.mountgetty.target.wants/ smartd.service snap-core24-1196.mount snap-snapd-25577.mount
using plocate eg;
locate xn62_f695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e
# returns
/root/.cache/kopia/2e424a7ff058859f/indexes/xn62_f695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e-c1.sndx
BACKUP SETTINGS: I am using the same settings on all servers:
{
“storage”: {
“type”: “filesystem”,
“config”: {
“path”: “/MYBACKUPS/DATA”,
“fileMode”: 384,
“dirMode”: 448,
“dirShards”: null
}
},
“caching”: {
“cacheDirectory”: “../../../../root/.cache/kopia/2e424a7ff058859f”,
“maxCacheSize”: 5242880000,
“maxMetadataCacheSize”: 5242880000,
“maxListCacheDuration”: 30
},
“hostname”: “myserverhostname”,
“username”: “root”,
“description”: “Repository in Filesystem: /MYBACKUPS/DATA”,
“enableActions”: false,
“formatBlobCacheDuration”: 900000000000
}
KEEP SETTINGS: Retaining ‘keep’ settings the same for all servers
–keep-latest=24 --keep-hourly=48 --keep-daily=7 --keep-weekly=1 --keep-monthly=1 --keep-annual=0"
BACKUP SCHEDULE: Using the same policy cron for all servers
default_policy_schedule=“0,30 6-23 * * 1-6”
ADDITIONAL FEEDBACK: Based on other posts, these may supply some added value
index epoch list command returns:
sudo kopia --config-file=$configs -p $password index epoch list
Current Epoch: 64
Epoch Started 2024-11-14 22:16:45 PST
64 2024-11-14 22:30:00 PST … 2025-12-06 14:34:08 PST, 92602 blobs, 221.7 MB, span 9280h4m7s
63 2024-11-12 22:30:00 PST … 2024-11-14 22:16:44 PST, 1067 blobs, 2.4 MB, span 47h46m43s
61: 2024-11-14 22:16:45 PST single-epoch 1 blobs, 1.4 MB
60: 2024-11-11 22:11:05 PST single-epoch 1 blobs, 2 MB
59: 2024-11-09 22:14:33 PST single-epoch 1 blobs, 1.5 MB
58: 2024-11-07 22:13:22 PST single-epoch 1 blobs, 1.2 MB
57: 2024-11-05 22:12:45 PST single-epoch 1 blobs, 1.4 MB
56: 2024-11-03 22:07:25 PST single-epoch 1 blobs, 1.5 MB
0-7: range, 1 blobs, 17.4 MB
8-15: range, 1 blobs, 2.7 MB
16-23: range, 1 blobs, 3.4 MB
24-31: range, 1 blobs, 3.7 MB
32-39: range, 1 blobs, 4.1 MB
40-47: range, 1 blobs, 4.6 MB
48-55: range, 1 blobs, 4.6 MB
REPOSITORY STATUS:
sudo kopia repository status --config-file=“$configs” -p “$password”
Config file: /SM_DATA/sm_backups/kopia/configs/internal.config
Description: Repository in Filesystem: /MYBACKUPS/DATA
Hostname: servermatter
Username: root
Read-only: false
Format blob cache: 15m0s
Storage type: filesystem
Storage capacity: 7.9 TB
Storage available: 1.1 TB
Storage config: {
“path”: “/MYBACKUPS/DATA”,
“fileMode”: 384,
“dirMode”: 448,
“dirShards”: null
}
Unique ID: 0ab1d81ee3f453fc9dc5bdf390eda2e26c3330c5e154d1e7b0ebb53fe1e838ae
Hash: BLAKE2B-256-128
Encryption: AES256-GCM-HMAC-SHA256
Splitter: DYNAMIC-4M-BUZHASH
Format version: 3
Content compression: true
Password changes: true
Max pack length: 21 MB
Index Format: v2
Epoch Manager: enabled
Current Epoch: 64
Epoch refresh frequency: 20m0s
Epoch advance on: 20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin: 4h0m0s
Epoch checkpoint every: 7 epochs
INDEX RECOVERY EFFORT: In the face of knowing nothing 
index recover seemed like a possible attempt to resolve:
sudo kopia --config-file=“$configs” -p “$password” index recover --advanced-commands=enabled
This took forever and went down a long list of ‘Recovered xxxx index from xxx blobs (found xxx blobs)’ for example:
Recovered 0 index entries from 0 blobs, estimating time remaining… (found 1 blobs)
Recovered 334 index entries from 40 blobs, estimating time remaining… (found 48 blobs)
Recovered 713 index entries from 82 blobs, estimating time remaining… (found 92 blobs)
Recovered 1194 index entries from 127 blobs, estimating time remaining… (found 129 blobs)
Recovered 1557 index entries from 162 blobs, estimating time remaining… (found 174 blobs)
Recovered 2191 index entries from 209 blobs, estimating time remaining… (found 213 blobs)
Recovered 2970 index entries from 228 blobs, estimating time remaining… (found 261 blobs)
Recovered 3358 index entries from 275 blobs, estimating time remaining… (found 296 blobs)
Recovered 3873 index entries from 323 blobs, estimating time remaining… (found 338 blobs)
Recovered 4507 index entries from 366 blobs, estimating time remaining… (found 372 blobs)
…18 hours later…
Recovered 1762041 index entries from 127356 blobs, estimating time remaining… (found 127359 blobs)
Recovered 1762698 index entries from 127397 blobs, estimating time remaining… (found 127402 blobs)
Recovered 1765323 index entries from 127414 blobs, estimating time remaining… (found 127462 blobs)
Recovered 1765919 index entries from 127461 blobs, estimating time remaining… (found 127499 blobs)
Recovered 1766318 index entries from 127502 blobs, estimating time remaining… (found 127536 blobs)
Recovered 1766903 index entries from 127552 blobs, estimating time remaining… (found 127574 blobs)
Recovered 1767450 index entries from 127603 blobs, estimating time remaining… (found 127617 blobs)
Recovered 1767996 index entries from 127651 blobs, estimating time remaining… (found 127654 blobs)
At this point there are no added messages in the terminal so I’m not sure if its taking a long time to continue to the next line after Recovered 1767996 index entries… or is completed and just giving no feedback. So I’ll probably just ctrl-c it there.
RADICAL HAIL MARY
In the absence of being able to dig up the real solution, I took some stabs at it knowing I still have a solid external off site backup in place.
I manually removed the problem child blob with
rm /MYBACKJUPS/DATA/xn6/2_f/695752f3cd84bdbaf216a8ead857503-s4c2b011519fdd9c312e-c1.f
I found out later there is a kopia delete blob command which would I’m sure have been a safer move.
I then ran kopia maintenance in hopes that it would just clean things up now that the problem child was gone. That resulted in “Finished full maintenance” but with a new content error with its own ID as shown below.
kopia maintenance run --full --config-file=$config_file -p “$password”
Running full maintenance…
Looking for active contents…
Looking for unreferenced contents…
… found 100000 unused contents so far (255.5 GB bytes)
… found 200000 unused contents so far (510.4 GB bytes)
… found 300000 unused contents so far (766.9 GB bytes)
… found 400000 unused contents so far (1 TB bytes)
… found 500000 unused contents so far (1.3 TB bytes)
… found 600000 unused contents so far (1.5 TB bytes)
… found 700000 unused contents so far (1.8 TB bytes)
… found 800000 unused contents so far (2 TB bytes)
… found 900000 unused contents so far (2.3 TB bytes)
… found 1000000 unused contents so far (2.6 TB bytes)
… found 1100000 unused contents so far (2.8 TB bytes)
… found 1200000 unused contents so far (2.8 TB bytes)
… found 1300000 unused contents so far (2.8 TB bytes)
… found 1400000 unused contents so far (2.8 TB bytes)
… found 1500000 unused contents so far (2.8 TB bytes)
… found 1600000 unused contents so far (2.8 TB bytes)
GC found 1642367 unused contents (2.8 TB)
GC found 6378 unused contents that are too recent to delete (13.5 GB)
GC found 679888 in-use contents (713 GB)
GC found 79962 in-use system-contents (1.1 GB)
Rewriting contents from short packs…
unable to rewrite content “444dd0527d3852682a7efa7b71a03ee4”: unable to get content data and info: error getting cached content from blob “p50563eb21a71d02c1be0f135fdbee9f0-s659403a1de3b18ae12e”: failed to get blob with ID p50563eb21a71d02c1be0f135fdbee9f0-s659403a1de3b18ae12e: invalid length
Total bytes rewritten 629.7 MB
Finished full maintenance.
ERROR error rewriting contents in short packs: failed to rewrite 1 contents
Upon attempting to start kopia now I’ve got some new problems relating to xn62_ as shown here
kopia repository connect filesystem --path=$backup_to --log-file=$log_file --config-file=$config_file -p “$password” --cache-directory=$cache_dir
ERROR unable to add xn62_db372a3db5cd5a516dc62b870228b8ce-sbbeb9f5f4b14009512e-c1 to index-blobs: context canceled
ERROR unable to add xn62_db4e821404261611f4ebd179b43c91a5-s04284aed00d2134712e-c1 to index-blobs: context canceled
ERROR unable to add xn62_dc23fd4a10dfe9d3496edc00eecb467a-sa46aa501e2dbb00612e-c1 to index-blobs: context canceled
ERROR unable to add xn62_d9998ca88382f2f107adbbd0a29fbe76-s81bd0cb2295f955f12e-c1 to index-blobs: context canceled
ERROR failed to open repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn62_dc4d4c839a16ff55537c04b07892a3fa-s659403a1de3b18ae12e-c1: decrypt blob: error decrypting BLOB xn62_dc4d4c839a16ff55537c04b07892a3fa-s659403a1de3b18ae12e-c1: ciphertext too short: 0
ERROR error connecting to repository: unable to create shared content manager: error loading indexes: error downloading indexes: error loading index blob xn62_dc4d4c839a16ff55537c04b07892a3fa-s659403a1de3b18ae12e-c1: decrypt blob: error decrypting BLOB xn62_dc4d4c839a16ff55537c04b07892a3fa-s659403a1de3b18ae12e-c1: ciphertext too short: 0
THIS DEEP SO GOING THE DISTANCE:
Decided to just delete those other complainers while I was in there
kopia blob delete xn62_db372a3db5cd5a516dc62b870228b8ce-sbbeb9f5f4b14009512e-c1 --config-file=$config_file -p “$password” --advanced-commands=enabled
root@servermatter:/etc/systemd/system# kopia blob delete xn62_db4e821404261611f4ebd179b43c91a5-s04284aed00d2134712e-c1 --config-file=$config_file -p “$password” --advanced-commands=enabled
root@servermatter:/etc/systemd/system# kopia blob delete xn62_dc23fd4a10dfe9d3496edc00eecb467a-sa46aa501e2dbb00612e-c1 --config-file=$config_file -p “$password” --advanced-commands=enabled
root@servermatter:/etc/systemd/system# kopia blob delete xn62_d9998ca88382f2f107adbbd0a29fbe76-s81bd0cb2295f955f12e-c1 --config-file=$config_file -p “$password” --advanced-commands=enabled
HAIL MARY FAILED:
When it was all said and done, I gave up, removed all the backed up files from the repo except the kopia.repository.f and I also copied kopia.blobcfg.f and kopia.maintenance.f to a safe location in case they every have a purpose.
I am now pushing the ansible role to this server and reinitiating the entire backup.
I feel doomed to repeat this process in the absence of knowing why it happens to begin with.
I do have other servers that would still benefit from the knowledge of a fix before I have to go this same route.
ADDITIONAL AI EFFORTS:
I’ve used AI till I’m blue in the face. Many false rabbit holes. Learned a lot about blobs and epoch’s but all to no avail. All dead ends at resolving the problem and knowing how I got there to being with.
This very important for me to discover as I am running many client servers and want to bring on more servers but need to understand this so I don’t “bake this issue into the server cake”
The backups appear to continue to work normally thus far. Yet continuing without maintenance seems inevitable that problems will come.
I have experienced one kopia failure on a server where I could not start the kopia service at all. A corrupted blob was the determined problem and I resolved it by manually moving the corrupted blob out of the directory structure eg; sudo mv /MYBACKUPS/DATA/xn7/_4d/1cf4964033f13e21dccb6a9048d16e-s1b365990ccd960f0139-c1.f /MYBACKUPS/DATA/corrupted_indexes
I eventually
What sayeth the group?
Jay - ServerMatter / CompuMatter
PS: added an issue on this in git issues but is 5 days old and no input yet. That is on another server but same symptoms Severe Index Bloat (28,000+ blobs) - Full Maintenance Not Reducing Count, Epoch System Blocking Deletion · Issue #5057 · kopia/kopia · GitHub