BLOB not found errors during snapshot verification and creation

Support
1

I’ve been trying Kopia as an alternative to restic over the last few weeks, backing up to an S3 bucket hosted on my local network by minio. Been liking it so far, particularly the compression and faster maintenance/GC.

Recently I’ve tried snapshot verification and and have come across lots and lots of errors such as:
[kopia/cli] failed on root@vserver:/mnt/vserver-backup-kopia-daily/libvirt-snapshot-mounts/discovery2@2020-08-31 11:51:50 BST/usr/lib/python3.8/site-packages/babel/locale-data: error reading kf8854ffe74b9c10bd1116cadc71cb122: unexpected content error: BLOB not found

I’ve tried a few things which have not helped, and possibly made things worse, such as clearing the cache and removing snapshots, but that doesn’t seem to make a difference.

I’m similarly now getting errors during snapshot creation:
20:26:46.276 [snapshotfs] unable to read previous directory entries: unexpected content error: BLOB not found

e: The snapshot command still seems to return a success exit status despite these errors.

Things that might have contributed:

  1. My network to the S3 server (on minio) has been unreliable and has failed lots of times during snapshot creation.
  2. I’ve been testing maintenance and have run lots of quick and full maintenance runs (on versions earlier than the current one – I know actual GC has been disabled recently)

Any ideas?

I’m not worried about losing the repository while I’m testing but, assuming those messages are a problem, I would like to understand the cause.

2

Can you DM me on Slack when you have some time so we can debug this interactively? I’m in PST time zone and evenings or early mornings work well for me.

In the meantime, can you stash away all the log files that Kopia produces?

3

Will do. Thanks for getting in touch.

4

Also, to preserve the repository state, can you do kopia repo sync-to filesystem --path <some-path> ?

5

Done.

(5959 BLOBs and 150GB)