Apparent compression header error on 0.9.7

Support
1

When running full maintenance, I receive the following error:

Rewriting contents from short packs...
unable to rewrite content "17d8ad0f1a5e691e1b0d0fc3d4760070": unable to get content data and info: error decompressing: invalid compression header, expected 00001102 but got 00001103

Followed by 20 others with different content hashes.

'kopia repository status' returns: 
Config file:         C:\Users\Michael Nickerson\AppData\Roaming\kopia\repository.config

Description:         Repository in Filesystem: \\serverbox2\mobilebackup\kopia\BIGWHITE
Hostname:            bigwhite
Username:            nickersonm
Read-only:           false
Format blob cache:   15m0s

Storage type:        filesystem
Storage config:      {
                       "path": "\\\\serverbox2\\mobilebackup\\kopia\\BIGWHITE",
                       "fileMode": 384,
                       "dirMode": 448,
                       "dirShards": null
                     }

Unique ID:           b33b031ca5cb2c07fc14f2d21309948211be894ba5efac0ed58f85146baace77
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
Format version:      2
Content compression: true
Password changes:    false
Max pack length:     20 MiB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 63

Epoch refresh frequency: 20m0s
Epoch advance on:        20 blobs or 10 MiB, minimum 6h0m0s
Epoch cleanup margin:    1h0m0s
Epoch checkpoint every:  7 epochs

I believe this error only started occurring once I upgraded to v0.9.7, but it’s possible it was introduced by a rearrangement of my backups: I changed from a single Windows client accessing a NAS via SMB shares to the Windows client only backing up local content and a Linux client on the NAS backing up the NAS contents. The change was concomitant with the upgrade to 0.9.7. I used kopia snapshot move-history to move around the various snapshot trees, and used KopiaUI to add identical policies to the new snapshot location (ignore list and zstd-best-compression only).

Please let me know if I can provide any more information or tests to debug this.

Additionally, is there any way to either repair my repository or delete the offending items? I do not mind losing the snapshots I’ve taken since upgrading to 0.9.7.

Thank you!

- nickersonm

2

This kind of bug happened already in 0.9.3 …

See also

Try to verify your content (100%).

3

Update: running kopia maintenance run --full --safety=none on the Linux client completes successfully!

So this seems to be something with the Windows client. Additionally, after running full maintenance on the Linux client, the Windows client no longer encounters any problems with full maintenance.

I have tried to reproduce this in a test repository:

  • Create repository on Windows, set compression policy, and snapshot a folder.
  • Connect to repository from Linux and move snapshot history to different location.
  • Set compression policy on new location via Kopia UI.
  • Snapshot new location on Linux.
  • Run full maintenance from Windows.

However, I was unable to recreate the issue.

Please let me know if it would be helpful to provide any additional information or tests.

- nickersonm

4

Did you verify the repo?

kopia content verify --download-percent=100
5

I just started that; it will take another 30 hours - 1.3TB repository on a a slow drive.

Edit: running a non-full verify first does result in errors:

kopia --config-file C:/Utilities/backup/extramobile.config content verify --parallel=1
Listing blobs...
  10000 blobs...
  20000 blobs...
  30000 blobs...
  40000 blobs...
  50000 blobs...
Listed 57913 blobs.
Verifying all contents...
  Verified 1 contents, 0 errors, estimating...
ERROR error content 0315933df5321eb1a7d729b3ec26fdcb out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 066f5d3ed5e9439db0c2c74410dff9cd out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 109871ac9e5927fc6d7e88bdc769fd90 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 127907a7ee5badf3aaff564e219dd643 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 136ca180cb1279bd988e0e4c45c80f73 out of bounds of its pack blob p5a59d30c8b026f286bb12a8bada08fcd-sbecf340813cd6951105
ERROR error content 15fa3a3c1bd5a13cf089db3c6ed02fb5 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 179a904baf2596fd7738ae87b55f6d6d out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 278f56fa9287b371b98e2ff76bb63058 out of bounds of its pack blob p5a59d30c8b026f286bb12a8bada08fcd-sbecf340813cd6951105
ERROR error content 27bdc6d96349f6c794dd54bd95c3b131 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 2941c7b1191185217aa335ac8bbd6de2 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 2a933c126318d35054f7503ea6b4e842 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 41fcc16bb6faa5863bab39f70de4a716 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 427aef3ee32913fc3c6ef661b1d976e9 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 4a4fbee59e45a312ccd061428ab2f402 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 6167e74ebd903c94d6ea7034d73f465b out of bounds of its pack blob p5a59d30c8b026f286bb12a8bada08fcd-sbecf340813cd6951105
ERROR error content 70c1e1bfacd67f9e993bc7623914f6b4 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 71881726051719335c606e7d53075de2 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 75f678a8d751a1698930fe328b441b82 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 78cfb8d1ba654b561682cf365142f828 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 7aa8623114cf784368b9ce0299ffe81c out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 7aacf1a0aa0b61f9e1768a428b580dc1 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 8674477bb2f67b8aebd8102b828335ed out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 88fb51bb749a12ba1f73291d5fc2fd04 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 8b581eacae04f1f53400e12698974fd4 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content 92844873e9331872c3828ba1e4bb7416 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content a19293e87c63efb4c53ee636225df4dc out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content b4a4fb50a33eb130fddd2081f672de4a out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content b7ad7e2a8fb2b6b6e34dec5b327bb55e out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content b9b60c7426a866d54b60f7565763a460 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content befafeba94d4fc58ffa839648b52cc2d out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content c0241c825d5009c07f645eecf788a7af out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content c87e3ca0548ba4b6c859dd3e8e663f62 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content c8b6ae1e152ee46a14b1a9a14beb0a3c out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content e26403c4ea37c838a0263bc0fb0a5522 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content e669626bf00d78dd43cb62a2e1c27a3b out of bounds of its pack blob p5a59d30c8b026f286bb12a8bada08fcd-sbecf340813cd6951105
ERROR error content ea928e84cdc40e8414470320575732fe out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content ef80a50992b228a19532685e54fc4f5f out of bounds of its pack blob p5a59d30c8b026f286bb12a8bada08fcd-sbecf340813cd6951105
ERROR error content f44c8a5111c1b5c9ec97ce82ace04617 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content f5a12dfa4f9b236f7c720c9c3aaa9f7c out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
  Verified 2436722 of 3082996 contents (79.0%), 39 errors, remaining 0s, ETA 2021-12-21 07:26:54 PST
ERROR error content f789a87a26dd1d2867bd14b8204f47f4 out of bounds of its pack blob p5a5968c293c3f992d649f59cc60a4e21-se14d418aa787be2f105
ERROR error content fcbe4e53910b8e8299afa74ceecf436a out of bounds of its pack blob p5a59d30c8b026f286bb12a8bada08fcd-sbecf340813cd6951105
Finished verifying 3082996 contents, found 41 errors.
ERROR encountered 41 errors

Is there a way to recover from these, or would it be best to recreate my repository?

6

A full verification (kopia --config-file C:/Utilities/backup/extramobile.config content verify --full --parallel=1) returned only those same 41 errors.

I don’t know if these are related to my original problem - should I create a new topic for the ‘out of bounds’ error?

Thanks for any ideas anyone has!

7

Update: the two referenced blobs, p5a59d… and p5a596… are both zero bytes. Is there any way to either recover these or determine which snapshots they belong to? If they are old versions of files, I would prefer to drop that snapshot rather than restart the entire backup from scratch.

Edit: I attempted to use kopia content delete to remove the contents I didn’t have valid blobs for, accepting the data loss. However, then kopia maintenance run --full completed with:

ERROR snapshot GC failure: error running snapshot gc: error iterating contents: Could not undelete referenced content: {a��+��♣�↓��↓�� 0315933df5321eb1a7d729b3ec26fdcb 0xc00037a480}: unable to get content data and info: error getting cached content: BLOB not found

Following this thread, I re-snapshotted everything with --force-hash=100. The full maintenance still did not complete successfully - it appears I am unlucky and an altered file was corrupted, so unless this issue is implemented I will continue having GC errors.

8

I was able to find the snapshots with the problems via kopia snapshot verify listing paths and dates missing the blobs, and comparing to snapshot times. Luckily the corrupted blobs were in snapshots of locations for which a snapshot history was unimportant, so I deleted the affected snapshots by a cycle of kopia snapshot delete and kopia snapshot verify until no missing objects were found. Full maintenance now completes successfully, as well.

It looks like this is the best solution for this kind of error until enhancement #799 is implemented.

For future reference, can kopia ever recover from filesystem or file-copy errors? Is erasure coding or similar planned?