Snapshots failed with "unable to complete PutBlob" message in logs

Checking snapshot status via Kopia server this morning, I discovered a lot of pending snapshot tasks together with these messages in the logs:

Error: error flushing writer: error flushing manifests: unable to write content: unable to create pending pack: unable to get session ID: unable to write session marker: unable to write session marker: sa3cd1f223784cfe2d06708352f5550ed-sd7eece2028c4c1db102: unable to complete PutBlob(sa3cd1f223784cfe2d06708352f5550ed-sd7eece2028c4c1db102) despite 10 retries.

and

Error: unable to determine if maintenance is required: error getting status: error reading schedule blob: unable to complete GetBlob(kopia.maintenance,0,-1) despite 10 retries.

While I was looking through the Tasks screen I saw Kopia automatically kick off a periodic maintenance task which completed successfully. The pending snapshot tasks then began to successfully execute.

The Tasks screen in Kopia server shows errors going back 6 hours. If I view the snapshot history for the failed snapshots and select “Show (n) identical snapshots” I can see a gap starting at 2300 yesterday for one snapshot, 0000 this morning for another snapshot. Snapshots are set to run hourly.

I am using s3 provider (Wasabi) and Kopia 0.8.0-beta3. https://status.wasabi.com shows 100% uptime for my storage region. My ISP’s service status pages don’t indicate any planned or unplanned downtime. Any ideas?

Can you join slack.kopia.io and ping me there so we can debug more? Are there any more logs in the log files on disk?

I can see these same errors in the cli-logs. What’s the best way to share these securely? We’re in different timezones so it’s difficult for me to provide realtime feedback.

One thing that jumps out from the logs is tons of stuff like this:

2021-03-26 01:04:34.759 W [repository.go:286] error refreshing repository: error refreshing content index: error listing index blobs: error listing indexes: error listing n blobs: error listing blobs: error listing all blobs: Get “https://s3.eu-central-1.wasabisys.com/[bucket-name]/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=n”: dial tcp 130.117.252.13:443: i/o timeout

The weird thing is that logging in to Kopia Server this morning to check snapshot status kicked off a maintenance task which seems to have caused snapshots to start working again.

This appears to be an intermittent network connection problem.

Anyway, can you dump your logs on https://transfer.jkowalski.net and DM me the link?