Snapshots failed with "unable to complete PutBlob" message in logs

Support
1

Checking snapshot status via Kopia server this morning, I discovered a lot of pending snapshot tasks together with these messages in the logs:

Error: error flushing writer: error flushing manifests: unable to write content: unable to create pending pack: unable to get session ID: unable to write session marker: unable to write session marker: sa3cd1f223784cfe2d06708352f5550ed-sd7eece2028c4c1db102: unable to complete PutBlob(sa3cd1f223784cfe2d06708352f5550ed-sd7eece2028c4c1db102) despite 10 retries.

and

Error: unable to determine if maintenance is required: error getting status: error reading schedule blob: unable to complete GetBlob(kopia.maintenance,0,-1) despite 10 retries.

While I was looking through the Tasks screen I saw Kopia automatically kick off a periodic maintenance task which completed successfully. The pending snapshot tasks then began to successfully execute.

The Tasks screen in Kopia server shows errors going back 6 hours. If I view the snapshot history for the failed snapshots and select “Show (n) identical snapshots” I can see a gap starting at 2300 yesterday for one snapshot, 0000 this morning for another snapshot. Snapshots are set to run hourly.

I am using s3 provider (Wasabi) and Kopia 0.8.0-beta3. https://status.wasabi.com shows 100% uptime for my storage region. My ISP’s service status pages don’t indicate any planned or unplanned downtime. Any ideas?

2

Can you join slack.kopia.io and ping me there so we can debug more? Are there any more logs in the log files on disk?

3

I can see these same errors in the cli-logs. What’s the best way to share these securely? We’re in different timezones so it’s difficult for me to provide realtime feedback.

4

One thing that jumps out from the logs is tons of stuff like this:

2021-03-26 01:04:34.759 W [repository.go:286] error refreshing repository: error refreshing content index: error listing index blobs: error listing indexes: error listing n blobs: error listing blobs: error listing all blobs: Get “https://s3.eu-central-1.wasabisys.com/[bucket-name]/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=n”: dial tcp 130.117.252.13:443: i/o timeout

5

The weird thing is that logging in to Kopia Server this morning to check snapshot status kicked off a maintenance task which seems to have caused snapshots to start working again.

6

This appears to be an intermittent network connection problem.

Anyway, can you dump your logs on https://transfer.jkowalski.net and DM me the link?