0.9.5 content verify --> error decompressing: decompression error: CRC check failed

Support
1

This is what I test

deleted all local kopia folders / clean slate
kopia repository create filesystem --path k:\kopia   (entered password)
kopia maintenance set --enable-full=false
kopia maintenance set --enable-quick=false
kopia cache set --content-cache-size-mb=1000
kopia cache set --metadata-cache-size-mb=1000
kopia policy set --global --ignore-cache-dirs=false 
kopia policy set --global --compression=zstd
kopia policy set --global --log-dir-snapshotted=0
kopia policy set --global --log-dir-ignored=10
kopia policy set --global --log-entry-snapshotted=0
kopia policy set --global --log-entry-ignored=10

Then created snapshots for multiple folders

kopia snapshot create C:\ProgramData --log-level=debug
kopia snapshot create C:\Program Files --log-level=debug
and so on
roughly 120GB

No errors

kopia maintenance run  (no errors)
kopia logs cleanup --max-age=1s

but after

kopia content verify --download-percent=100

I saw these errors

  Verified 14775 of 294807 contents (5.0%), 0 errors, remaining 44m44s, ETA 2021-11-22 00:56:18 PST
ERROR error content 0eff080225ddb9bcb1be382257e48471 is invalid: error decompressing: decompression error: CRC check failed
  Verified 14986 of 294807 contents (5.1%), 1 errors, remaining 45m0s, ETA 2021-11-22 00:56:37 PST
...
  Verified 85885 of 294807 contents (29.1%), 1 errors, remaining 33m21s, ETA 2021-11-22 00:56:16 PST
ERROR error content 571d75669a0b2cae7c05f46dd17adb41 is invalid: error decompressing: decompression error: CRC check failed
  Verified 86154 of 294807 contents (29.2%), 2 errors, remaining 33m19s, ETA 2021-11-22 00:56:17 PST
  Verified 86396 of 294807 contents (29.3%), 2 errors, remaining 33m19s, ETA 2021-11-22 00:56:20 PST
...
  Verified 87667 of 294807 contents (29.7%), 2 errors, remaining 33m6s, ETA 2021-11-22 00:56:19 PST
ERROR error content 58dd969485b9dc780fdcbf6dfbd25d1c is invalid: error decompressing: decompression error: CRC check failed
  Verified 88030 of 294807 contents (29.9%), 3 errors, remaining 33m2s, ETA 2021-11-22 00:56:18 PST
  Verified 88385 of 294807 contents (30.0%), 3 errors, remaining 32m58s, ETA 2021-11-22 00:56:17 PST
...
Finished verifying 294807 contents, found 3 errors.
ERROR encountered 3 errors
2

Since you also had the problem of zstd suddenly getting mixed up with gzip and it resolved itself - i wonder if there’s any hardware issue here (RAM for example?)

Can you:

  • wait and try again?
  • reboot and verify again?
  • use another machine to connect to the repo and verify it?

See if you can find a pattern there… If you can consistenly reproduce this (even if different behavior every time), please ping me on Slack so we can work on fixing it.

3

I don’t think it is a RAM issue. Regardless … I tested the RAM (it is ok). And I verified the repository on a different machine and Linux instead of Win. But I also got CRC errors there.

I don’t see any pattern.

4

Can you run it a few times (10 perhaps?) on fresh repository each time to see if this reproduces?

5

I don’t know what you mean by a “fresh repository”? (a fresh repo in my vocabulary means “newly created repo without data”). In my testing I create and delete repos “daily”. I have run it 10 times (it takes a lot of times …), but I did it a quite few times and got crc errors (but not always). It does not happen on smaller repos.

I would suggest you try the same. Create a new repo, enable zstd, snapshot 150GB files to a filesystem repo. Do a full maintenance run, and then run a 100% content verify.

I am also wondering, is there a way to find out to which files(s) the content blocks with crc errors belong? Maybe there is a pattern there?

BTW, I also verified backups created with other tools, but there never has been an issue.

6

I ran this serveral times on my windows box using a script, but it was not able to reproduce it, I’ll keep trying but I would appreciate if you could help me narrow it down:

  1. Are those large files, small files? Generally compressible or non-compressible?

  2. Would you be able to try with a different compression method and see if you can reproduce it? Also without compression?

  3. Also, I would like to double check that there is no activity between snapshot and maintenance? Maintenance after a single snapshot should basically be a no-op since there is no data to rewrite or delete. Wondering if you can reproduce this without maintenance at all?

7

I did the test with gzip a few times before but there no were no crc errors. But I ran it again today.

I noticed one issue. zstd is supposed to be fast, certainly not much slower than gzip. However, verifying a repo with zstd takes 45 minutes, while the gzip repo only takes 15 minutes. Do you see the same in your tests?

There is also another weird issue I noticed

zstd verify:

Finished verifying 294807 contents, found 6 errors.

gzip verify

Finished verifying 200859 contents, found 0 errors

There is a huge difference in the number of contents. Do you see the same in your tests?

The zstd repo is 3GB smaller than gzip which is to be expected since zstd has better compression.

The source is a mix of large, medium, and small files, roughly 50% is not compressible (according to content stats).

There was no activity between - other than snapshot list or so (and I disabled automatic maintenance as well).

8

Hi, were you able to find the culprit of the issue?

~Horo

9

No, it seems to be a bug in Kopia.

Unfortunately the developer has abandoned this question … (Even though I raised interesting points in my last reply: 1. Number of contents. 2. zstd slower than gzip …)

My recommendation is to stick with the default compressor gzip. It seems much more stable.

10

I see, thank you for your reply!
When I manage to get the time I’ll try to recreate the bug.