Repeated 100% snapshot verify prints out different "error getting cached content from blob"

Support
1

Hi Folks;

I’m running Kopia CLI on a Ubuntu 22.04.5 LTS server, and the Kopia repository is hosted on a Wasabi S3 data bucket.- the specific CLI build is:

            0.21.1 build: 0733cb4d2a731dbb92d927f66230694e014f4df2

After doing an initial large test backup, I used the following verify command to check the backup’s integrity.

            $ sudo time kopia snapshot verify  --verify-files-percent=100 --file-parallelism=10 --parallel=10 --log-level=error --file-log-level=debug dab4b6a154458b782d71a74d5abbd537

That produced errors for six files - here is a typical error message:

2025-11-26T15:50:28.847126Z ERROR repofs error processing root@velvet:/home@2025-11-24 13:48:37 EST/fred/Projects/test.raw: error reading object Ix796d4d1feb313f3c0924299ef9
cadb71: unable to read data: unexpected content error: error getting cached content from blob “pd87eefbe6605c20943f46d9c
8eed7929-sfe7806906dd1afd813b”: failed to get blob with ID pd87eefbe6605c20943f46d9c8eed7929-sfe7806906dd1afd813b: unabl
e to complete GetBlob(pd87eefbe6605c20943f46d9c8eed7929-sfe7806906dd1afd813b,2253833,122122) despite 10 retries: unexpec
ted EOF

I then re-ran the same command, and got similar errors on two different files - the initial six files did not appear in the error list.

I did test restores on all identified files from the two runs, and used diff to compare the restored files to the current server files. Found that a large (~80GB) .tar.gz file was different (via diff and sha256sum) AND I could not untar the file.

I am currently running this command to fix the repo:

$ sudo time kopia snapshot fix invalid-files --verify-files-percent=100 --parallel=10 --log-level=error --file-log-level=debug

After this I’ll run another 100% verify to see what happens.

Any thoughts on what might cause the verification errors?

Thanks!