0.9.5 content verify -->> unable to get content data and info: error decompressing: invalid compression header

Support
1

I snapshotted a few folders (~100GB) into a filesystem repo (newly created, zstd enabled).

After that I ran

kopia maintenance run --full --safety=none

but got the output below.

kopia content verify --download-percent=100
Listing blobs...
Listed 3878 blobs.
Verifying all contents...
  Verified 1 contents, 0 errors, estimating...
ERROR error content 0002dd26b934b9707fdbf30341e2f6cc is invalid: error decompressing: invalid compression header, expected 00001100 but got 00001000
ERROR error content 00069853f583aee900005064e55e6edb is invalid: error decompressing: invalid compression header, expected 00001100 but got 00001000
ERROR error content 000a2ae81be20026f84fd66a6249652b is invalid: error decompressing: invalid compression header, expected 00001100 but got 00001000
ERROR error content 0014a62961e70bde5b5aae0ddb45693e is invalid: error decompressing: invalid compression header, expected 00001100 but got 00001000
ERROR error content 0017cfcf684c05f19646cf60893646e5 is invalid: error decompressing: invalid compression header, expected 00001100 but got 00001000

AND HUNDREDS OF SIMILAR LINES FOLLOW

What went wrong here?


Rewriting contents from short packs...
unable to rewrite content "021bd1c63f0d858d6d303b74b53b2431": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "07f03a75922112069c9414d219a4dd69": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "04f3757fa708a71dda7f5471958e5c87": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "03bef20f1f350256886ed48380d3b761": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "0182798b3096285f4be6fe1d04d0b6f3": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "07ca7feee895280a2a1a8cc9d556b26d": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "0a6c34562212472f6fed06a3f86ed1a7": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "0b49a5fb7f9eaf28ba392b4512d47600": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "08a2d214686b4800172d164b30473319": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "0881dd3588d93c0da4f6ac1f25ca5487": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "087b5af033dbd6a4c22f47981bfad314": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "0c07e0c7e8a8b501dc8609f7b60ce955": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "11ee0c50a7c26281cfc647f3b55c6379": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1584ea9c034bc8a6b4ccac22510a09c5": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "161e232972af1c05e4d2f945ad448d2f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "15a1db91f93f5af6b755e97e18cba85a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "16df510cba2bb24688b7bc7ad58011dd": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1a4fcd86161767092c261ded0318c6b9": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1a506dcd20411b62e8b1e1ac60fa5233": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "17a484b5d6d7ad3f3c73f6506a43f767": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1b5746b8e241e7e1bfabc5623b95ea19": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1d5c6ed07245b3d569a6c55a067f8f39": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1dfcc1df64cfd76dea40f78dfcc7597d": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1ac380bc1696bfb2bb35af2d1be48d5f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "204c74b96395922727c3e7682011f8f1": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "1ede459276f7aee4aacceee7209e3795": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "2024c0ecde7bb002dedd23f435b0ae72": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "251847b014ded740bfb7a11b72f7f09b": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "296272dd95d6b94c40be3676575a6252": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "27883f6d0166066eb892437490c6b1ee": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "29e5e671abbd0512912522ae66e47bb0": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "2b23a1218a7f363aa789179c4789ab95": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "2aa983f46d99f555fadf010229556fae": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "2b8950cc850344aa58f05712b00397a5": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "2b9a5aa4a964b9aeca610e4abb35ee34": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "2e6c9cda9972e9d889900659d4b586e7": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "322c46a32b1a67c9b19ca37f09d9d31f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "32a4f28d86f00ca809772ff024065565": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "3552920e06fc6a3d8fec1ba88fadc988": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "35f3702553094bdfe75b56d3f61bc25a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "36c364a42b86733db54d0b09782d6e95": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "3863eadef76126a25073a4a5ddde567c": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "3aab94db043b120d6660c10ad78239ee": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "3c3feed91f131f62338e651dca425b72": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "3c78a04512a567276d8eb097110f6813": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "400edfde81aabf29f5ca0443f7578e42": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "43cbb92cc20ef2c323bf9dce9c5cd837": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "441274f5ec47725e674a2e79856e236c": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "43df1adc93d2b393f1181bef665c57a4": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "44f9bb326360b9e37c10c498f16419fa": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "45f7afc1dd2cac6f2b5dcd92106141d5": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "4bc25e4166d775e3b80db0bcc5f129e7": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "4b0e26c8ba1069ab7930a61374cc18fd": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "4e3b64e9be023ea4d27c0d839c607c41": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "4eabf1a63fea34ea4a638f2f23e77209": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5b1dae665310b740a40952cd9170ed9a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "523ed63517068937691764f76dbe5163": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "4e975b545dc2e231d61ec66f6554a804": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "4f5410dadd899b7c00d3e8533b28e786": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5649028d12f4e8597cd094957ac342eb": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5565c92c19b96bad7356af555725ee74": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "54fbf305f9e7a20c6e35e453236ea42c": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5e9efdb2d86107d39c715791bfbbd92d": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5f81b05756df1b112b638935e211356c": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "65520b4f440e71efb65c40bbf00707a2": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5c44efbac2618766dd2a3d75f2a7b425": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "60162d2bce581c3f442e59bcc31a21b0": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5e8ed917bad58691af69dcc648b6d520": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "638a6489ebdcba32e911aae536265564": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "5c9c7a8e0e57781efc2279c6bb9a5dfc": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6a0e21b4c91a1e002cb55c54b03b6455": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6914f2c8c71913db028944c5c945aa9d": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "66372603f5d27e043e67f7d9f4c5e07b": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6b8bbf377eeb703303e6348e33debe3f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6734ae59857cbf7259c2a8a57ddb90b2": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6ba80e7fc89d1cfe85d633c3c8a20d0f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6dc0465c2cff14cfafc0a05e0d86be3a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "70773402812a34131de6f1c688dbaba3": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "70a22a53829774d2c1880c6d6feab0ec": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6c051cda18acc0ced4337b59fc78c2f4": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "6dfb2079c64251d7b854e39c3fbc6632": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "7801c4866b169d8b4f2bfdf5f16b5ab0": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "7298360495121e9d40ac7cb547202a9b": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "73bef9beadebdfb7eb72b527dc7afae8": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "796fbb75633b21c317f4a89f4ffeb3e2": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "7a6b1a7a69ae660c1357a5d1d7b94f73": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "82996143a763c480933be076768de217": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "8059d7d8cbbc6059c03e7ee53cca6cbc": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "8114ceb90520c05d59abed0a62b57222": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "82adcdd1c4d4a6b506ebc3da550ac1b1": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "8532a5d6f19a91d86baef781271439ec": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "85857a16c59a20feca8a5ea132184fdd": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "861e6bafcfdd6faa15e7c5b291e7ad68": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "85cd520a34dae89efcaf0d3cc069c743": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "878c1a6ad7d210adc2a4a62eaa8d4864": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "8783dae0fc92d909bd31d536d44e9b61": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "9002cc20384222058116788314d36585": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "975eb6c0bb4e1f10217a67a3be489f96": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "953637c8f19fb8236f6c6a834e2e156e": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "95dfc709c81079f6043726a164a3b647": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "98dd34d26291018e9a7159ef6591d5cc": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "986fc57d0a161ff215c740663136675f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "910c7f8245d716b14dabe42e54385c7f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "95532e236ff257b759f9de95994c4d8e": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "91de1ade29116c6b24ca2cf1bb92a605": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "996cd0a429b59f44a933651f91f0dc3e": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "99decea9ad72c11c342436f1769faadc": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "94c26f230549050ee671db9873830dc2": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "9d6c3c54c8f0587e922478740373764b": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "99d97179199e17b0076818729ca55aed": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "a9c2a18bda5ad0b5f014c9c8b5621518": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "a7fd92d3c06e89ac1bb87c0c1e2db0b6": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "a63ab61e1dfe404b6140d36001f08cea": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "a3130fb09454c035c523ecb49637f7ff": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ad3fdb214a0d661a56308220cdcf4919": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ab3cd5d2b50016bddb6b61dd00954e8f": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "aa1ebd7199212d916ac3dd9788fbdfa2": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ac120ce569acea227c8af66c215b9f45": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ab2954e6094575d6318bd89c1b052d49": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ad572500919346f511221a46b624c08a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ae1e450791bd341912c50f0c90a54a30": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "afacdbffb1f8d857f13e82cd90e16cea": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ae521190a8fedc6fb5c8cfaf10ef8577": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "afcf7257d05413ff1e4c15870f6df590": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "b0ce2aef182f4d32fb0b16c9d346ca59": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "b0404f441c9aae07cf49e7c69f4f31b5": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "b2abaa66dcd8e874d92b393c48e6c55a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "b48e65aa07c8ae4c8222f00fe7a2c352": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "b44528f951b9a5818f318f0bc00d886a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "b2d987d43a03a152e0bd9803f3f01440": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ba9c2f538a9713aec05c469f58ec1f4c": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "bd88ad1d0cfa5ff27bdb23c5b99fbcc8": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "c0700dd3cc02ba1e1f29a2050a9441ed": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "c10be9be71909f84199512c3f4ca239c": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "c388ec8e4788f3a917b38d3f810f0d93": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "c7c19e034eaff1394cd52f8e230f8e2d": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "cdf0cd752789a7a1de9dc48bccf21dff": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ca2f3b39c366a52d60fc8f62b124f6ed": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ce7276436d404e3a3b047f1a8c69feb9": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "c951df164c3622488bf8931adcb4c5b1": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "cb01a1c3359b36ec9b8452fe4dcbe12e": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "caff1a6ccf0e8d8c822c471ed76890d5": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "c90981d7f1b8d25a7ac235edb5eb1960": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "cc3d9a2aa3376760a0891b5f6f15d8a2": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "d26ab7e9e05eca58e2886c8f80fc9fbb": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "cfe030f7565122f265eb011472d3afa5": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "cf0cca5777e10fa18914e2c7701694a4": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "d3ca4a9547f78075e86d4e62d920c50b": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "d317f4596826cf3b74d82fd036f5bd2c": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "d3420df100ce7d70c2c1c21173a374d3": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "e29c6d37637ac1207080e3a60b2cab1a": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "eb6694285447edebd4d71149813265d9": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "e6630a2e3515eed0bf0e6ae6bf046cd7": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "e278197101c39923f67bd62517d6f534": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "e103bf6d464c21f879454e528defc1cd": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "f0da077d3cf5740c0bcfd9684fe9cb65": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "f039c6624b9bf95e10e8a31119976601": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ee5b0fe51960346967a6670bfa685c59": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "eccd39a2afd1f950646e89e86e092366": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "ee26f506b1f47c6dc210958358a1dc6e": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "f49a6046d45cc83a0cb2be8d718c811b": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "f145d641546df24ab83364d7869d3bf9": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "fa59474446850ea5010590b1ab16ab53": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
unable to rewrite content "fa3516a64aa2301c5e7710def5cd1a35": unable to get content data and info: error decompressing: invalid compression header, expected 00001100 but got 00001000
Finished full maintenance.
ERROR error rewriting contents in short packs: failed to rewrite 164 contents
2

That’s very interesting. Can you file a bug on GitHub? Are you able to reproduce this on a clean repository?

3

00001000 is GZIP default
00001100 is ZSTD default

and they must have gotten mixed up somehow in the maintenance/rewrite process. I would appreciate if you could reconstruct the history of actions on this repository to see how you landed in this state.

4

I am not able to reconstruct it. Actually I ran another kopia content verify on the same repo (unchanged since the last verify) and there was no error …

kopia content verify --download-percent=100
Listing blobs...
Listed 3814 blobs.
Verifying all contents...
  Verified 1 contents, 0 errors, estimating...
  Verified 275 of 303035 contents (0.1%), 0 errors, remaining 55m18s, ETA 2021-11-19 20:31:38 PST
...
...
  Verified 293853 of 303035 contents (97.0%), 0 errors, remaining 1m16s, ETA 2021-11-19 20:18:21 PST
Finished verifying 303035 contents, found 0 errors.

How is this possible?

5

And I just remembered that I had a similar issue a while ago:

error decompressing: invalid compression header, expected 00001000 but got 2320426c
Finished full maintenance.
ERROR error rewriting contents in short packs: failed to rewrite 1 contents

Not sure what 2320426c is?

https://kopia.discourse.group/t/v0-9-3-compression-error-error-rewriting-contents-in-short-packs-failed-to-rewrite-1-contents/688

6

The behavior case suggests some caching issue (likely a bug), the second - some other kind of corruption. Given that I’ve not seen this in any other case makes me suspect there’s something particular about the environment you’re running this in.

Is this the same repository?

7

This is not the same repository (already deleted). I usually delete the repository and create a new one. Although probably I do delete the cache less often than I delete a repository :slight_smile: I am still in the testing phase :wink:

Compression has quite a few “weird issues”.

For example after enabling compression, I usually see output like:

kopia content stats
Count: 303035
Total Bytes: 116.8 GB
Total Packed: 86.3 GB (compression 26.1%)
By Method:
  (uncompressed)         count: 103525 size: 51.6 GB
  zstd                   count: 199510 size: 65.1 GB packed: 34.7 GB compression: 46.8%

It is not clear to me why roughly 50% is "uncompressed’? There should not be any method “uncompressed” because compression had been enabled before snapshotting anything.

It is difficult for me to create a simple “repro” example, since the outcome often is not predictable (as you could see with the verify example. The first verify had many errors, second verify no errors …). I suspect there must be some kind of race condition?

8

“Uncompressed” usually means the compression was attempted but data was not compressible.

9

“Uncompressed” usually means the compression was attempted but data was not compressible.

Ok, I understand. But I would probably prefer still “compressed” or maybe better “attempted compression”.

10

I am getting something similar in 0.10.4 in Windows:

Looking for unreferenced contents...
... found 100000 unused contents so far (37.8 GiB bytes)
... found 200000 unused contents so far (64.1 GiB bytes)
Rewriting contents from short packs...
unable to rewrite content "bec655787ad2fd7f3af22b66969abf4c": unable to get content data and info: error decompressing: invalid compression header, expected 00001203 but got 00001101
unable to rewrite content "b8baac38ec65ed10a42d18cca77fd0f6": unable to get content data and info: error decompressing: invalid compression header, expected 00001203 but got 00001101
Finished full maintenance.
ERROR error rewriting contents in short packs: failed to rewrite 2 contents

Unable to do anything… content verify not working, not sure how to fix this issue.

please let me know if there’s a solution to this issue. thanks!

11

I was actually able to run the maintenance from the storage’s host (synology), using kopia CLI vs running the maintenance from the typical owner, my Windows 10 machine connected to the remote repo with SMB.

Therefore, per the above suggestion from Jarek, does this mean this is a cache issue somehow on the Windows 10 machine? Is there a flag to run maintenance and ‘force it to ignore cache’ ? Alternatively, is there a cache purge option short of deleting the actual files from the cache folder?

Now that the maintenance ran successfully once, I will reset the owner to the original one, and run snapshots, maintenance, etc… again from the main client, but I am curious what this means for the future.

12

FYI I got similar report on Slack today and now have a solid theory on what may be going on. It’s likely related to data caching, index merges and content rewrite which runs as part of maintenance.

I’m hoping to get a clean repro and will make a fix.

13

Good news, the bug is now understood and the fix is under review:

I’ll cut a new build in a few days after this is merged and we give it a chance to run for a while.