Hi!
My daily kopia snapshot verify --verify-files-percent=1 failed today for one of my 8 repos. 4 repos are stored locally on a hard drive. The other 4 are on Backblaze.
The error happened for the Backblaze repo:
ERROR error processing kopia@kopia:/mnt/source@2024-07-12 20:36:14 CEST/Kamera/blahblah/IMG_1651.jpg: error reading object 81b395c424ead2bf01f00cbb97ac722a: unable to open object 81b395c424ead2bf01f00cbb97ac722a: unexpected content error: error getting cached content from blob "p2adfb903313698128cad134edb15efb3-s37cb845dd12e8de412a": failed to get blob with ID p2adfb903313698128cad134edb15efb3-s37cb845dd12e8de412a: unable to complete GetBlob(p2adfb903313698128cad134edb15efb3-s37cb845dd12e8de412a,14736024,2736641) despite 10 retries: DownloadFileRangeByName: internal_error: incident id bf14856d5f89-e8ddfca8f68da5db
All other checks were successful. I just ran a verification manually and it failed again for another file with the same error. Then I located that file in the web UI and successfully downloaded/restored it. So my guess is, nothing is broken or corrupted in the end. I guess the connection to Backblaze is just flaky. I never had these problems before since I built this backup solution (like 9 months ago).
Unfortunately my Backblaze account stores everything in the US because I misclicked when creating the account. I am living in Europe. I suspect it might contribute to the issue.
Did anyone have any of these problems? What would you suggest to do?
If you are using native B2 connectivity try to connect using S3. Or vice versa.
I’ll first wait for some days to see if the issue persists and then test your suggestion. Also this issue reminded me that the data is stored in the US. I might switch to Hetzner’s S3 storage once it’s released.
I think it was released yesterday.
From my experience S3 is the best option for kopia.
1 Like
Great, I just have applied for the beta 
Getting accepted to the beta program took at max one day. On 2024-11-01 it will be made generally available. I have just tested to backup like 1 GB of git repos and restored it. The internal kopia provider validation reported no errors. Of course this is just a small test to just verify that all functions that are needed by kopia do work. I will do a full test with all my data in the next days I guess.
1 Like
I also have connection issues to Backblaze and the same thing happens using B2 API or the S3 compatible API. In my case the error I get on KopiaUI is: unable to complete GetBlob(kopia.repository,0,-1) despite 10 retries: Get "https://s3.eu-central-003.backblazeb2.com/MY_BUCKET/kopia.repository": dial tcp: lookup s3.eu-central-003.backblazeb2.com: no such host
However connecting via Cyberduck and downloading any file such as the root kopia.repository file works correctly. Any idea on what to try on KopiaUI side? Bacblaze says that as it works with Cyberduck it must be a KopiaUI issue.
I guess your problem sounds more like a DNS issue. Can you try different DNS?
Btw Hetzner a S3 works great. I have uploaded something like 1 TB and daily connections work great.
I have tried different DNS settings in both my router and MacOS settings of my wired network. I have also tried using completely different networks with different DNS such as the mobile hotspot from my phone. Doing a ping to s3.eu-central-003.backblazeb2.com from the console always works correctly. Cyberduck has no issues downloading and uploading files from the same bucket on the same network.
golang (what kopia is coded in) networking has its quirks - try to connect using rclone (also golang but has more config options) and force IPv4 + disable HTTP2. --disable-http2 and --bind IPv4 are flags to try.
In case if it works we would know where to look at for potential proper solution.