Tracking down backup errors

Support
1

For one of my backups I get this:
2020-09-19 02:06:31 EDT k96835e856c79785e0ad26a0e187ce512 175.6 GB drwxr-xr-x files:538285 dirs:47040 errors:2 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

So I used kopia ls -l recursively to get to the last folder with error. For that last folder it doesn’t show any errors. So how does one know what failed? Have to go to logs?

Showing below the folder right before and then the actual folder with error

kopia ls -l kaa0e54e9868f5187c24a1a95274f57c7|grep error
drwxr-xr-x 115790222687 2020-09-19 02:13:51 EDT k1c7260abcc0b8413cd37e6695011743f  postgres/ (2 errors)

kopia ls -l k1c7260abcc0b8413cd37e6695011743f
-rw-------   1147666432 2020-09-19 02:13:51 EDT Ixeb5332b53c3a298e3c0f57a985948647 .cz-Sat-00.pgdump.eQQ9rn
.. other entries - no errors ..
-rw-r--r--       457372 2020-09-18 13:29:47 EDT 940e569aa501d9313bdd27f1ff4b5f96   pgbackup.log
2

This is what I found on the log for “Error”

2020-09-19 02:14:15.743 D [logging_storage.go:46] [STORAGE] PutBlob(“p7b65a7bd38b605c293f0d55469e839e6”,len=23810437)=&url.Error{Op:“Post”, URL:“https://pod-000-1008-16.backblaze.com/b2api/v1/b2_upload_file/07d987028e6b6c3d7c48001e/c001_v0001008_t0031”, Err:(*errors.errorString)(0xc0000847c0)} took 122.535633ms
2020-09-19 02:38:40.443 D [logging_storage.go:46] [STORAGE] PutBlob(“p1abbf009a734b4a7a2aab4b217f9e462”,len=21348667)=&url.Error{Op:“Post”, URL:“https://pod-000-1144-17.backblaze.com/b2api/v1/b2_upload_file/07d987028e6b6c3d7c48001e/c001_v0001144_t0054”, Err:(*errors.errorString)(0xc000084030)} took 8m27.215161961s
2020-09-19 02:39:00.184 D [upload.go:739] finished reading directory czdata/images/cz/thumb/3/3a/NFPA_Template_Firefox_Error.png in 117.311µs
2020-09-19 02:39:04.622 D [upload.go:739] finished reading directory czdata/images/cz/thumb/d/d0/Errorsubpagecollapse.jpg in 103.123µs
2020-09-19 02:39:24.368 D [logging_storage.go:55] [STORAGE] DeleteBlob(“n0bf4b5bfc2b9a7a3cf3e5654231dbc2e”)=&backblaze.B2Error{Code:“no_such_file”, Message:“File not present: n0bf4b5bfc2b9a7a3cf3e5654231dbc2e”, Status:400} took 74.199846ms

3

The error details are not easy to see, but here are couple things to try:

  1. kopia server --ui to bring the UI, then click on the URL it prints and see the error in the browser
  2. kopia show kaa0e54e9868f5187c24a1a95274f57c7 - will have the error in JSON near the end
  3. kopia snap ls -m to find the manifest ID of the failed snapshot, then kopia manifest show <manifest-id> will have the error.
4

Intalled ltest 0.7

kopia --version
0.7.0 build: d0d6ac4767cff07ba080e2ef9eeb03edc082abc5

Ran kopia snap ls to get ID of run with error

root@storage:~#kopia snap ls -m k828e55b58b69147910029b5e9dcd334e
root@storage:~#

Got no output. Tried 4 others. Same, no output from ls -m

5

kopia show did work. Any reason we have the output as Json and not ins some more human friendly format?

[edit]
Output from kopia show doesn’t really give much info on why file failed.

 'errors' => 
        array (
          0 => 
          array (
            'path' => 'czdata/backups/postgres/cz-Tue-12.pgdump',
            'error' => 'Post "https://api001.backblazeb2.com/b2api/v1/b2_get_upload_url": EOF',
6

looks like this was a B2 service issue (the error was EOF), we probably need better retry policy in the provider.

FWIW folks are having better success connecting to B2 using S3 provider instead.

7

Understood, but is it expected to fail every single time? Just checked and my last 7 daily backups have had one or two files fail.

if B2 support is this unreliable should we perhaps consider outright removing it? Since B2 now supports S3.

8

I’m using the UI to monitor snapshot progress and was also wondering what were the 25 errors counted on the badge on the task details page (screenshot omitted, new user restriction)

It turns out that you can see the errors in a tooltip by hovering over the warning icon in the snapshot list:

Screenshot-2021-11-14-18:06
Screenshot-2021-11-14-18:06579×632 31.4 KB

P.S.
I’m trying out Kopia for my home lab backups, and am so far very impressed with the feature set, and the level of thought that went into the implementation. Looking forward to using this to keep data safe into the future, thank you!

9

As the error says those are unsupported file types (sockets from what I can tell). Kopia does not support snapshotting sockets (yet?).

2 Likes