A few other tests I did to try to pin down what is happening:
(1) I tried doing the restore but without writing atomically. I stopped the restore when it threw the same error for the same file in my OP. I did not complete the restore (because it takes about 12 hours), so I am not sure if the errors for the other files would have persisted, but it looks like the issue may not be with the atomic
library.
(2) Just for testing purposes, I create a new policy of just the one folder that contains the file that keeps erroring out. The idea here is to allow quick restores to try to pin down the issue, rather than having to wait hours. To my surprise, the problematic file restored just fine when restoring the test snapshot, without any errors. I tried this both atomically and not atomically. Same result.
I am not quite sure what is up, because I know the files that threw errors are not corrupt in my original snapshot. I’ve run snapshot verify --verify-files-percent=100
and I am able to successfully download the individual files from the snapshot. But the fact that the file restores fine after I created a new policy is weird. If it helps, the original policy backs up about 1TB of files. The new test policy only backed up about 100 MB of data.
(3) I deleted all my snapshots for my original policy but did not delete the underlying blobs (so Kopia will not reupload the data). I then rerun the snapshot and did the restore again. It started throwing the same errors for the same files as before, so I stopped the restore.