Can't open repo, manifest corrupted and BLOB not found repo

Alejandro_Del_Rincon · August 28, 2023, 3:59pm

Having this error using Kopia on windows on a NTFS USB-attached drive:

Connect Error: INTERNAL: internal server error: connect error: error connecting to repository: unable to list sources: unable to find manifest entries: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID qdb3b915a20cc804d0e38a8c11317dcad-scfce0e8f6057575d11f: BLOB not found

For some reason, this is like the 2-3rd time it happened to me using the exact same setup (I still think Kopia breaks itself when doing maintenances).

Seems I am not the only one…

github.com/kopia/kopia

ERROR: unable to get maintenance params: error looking for maintenance manifest: unable to load manifest contents: error loading manifest content: getCacheForContentID: BLOB not found

opened 12:19AM - 20 May 21 UTC

closed 08:20PM - 07 May 23 UTC

dbousamra

stale

My repository seems to be in a corrupted state somehow. I've done nothing other …than periodically run snapshot create. If I do a `kopia content verify`: ``` Finished verifying 1222 contents, found 272 errors. ERROR: encountered 272 errors ``` What do I do next?

github.com/kopia/kopia

corrupt repository? error connecting to repository [...] BLOB not found

opened 07:54AM - 01 May 23 UTC

probablykasper

Shortly after renaming my macOS boot drive, I got this error in KopiaUI: > Er…ror: error connecting to repository: unable to list sources: unable to find manifest entries: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found. <details> <summary>Logs</summary> ``` 09:38:08.357 unable to check for updates: unable to open update state file: open /Users/k/Library/Application Support/kopia/repository.config.update-info.json: no such file or directory -- 09:38:08.388 password for /Users/k/Library/Application Support/kopia/repository.config retrieved from OS keyring 09:38:08.390 generating new TLS certificate 09:38:08.656 adding alternative IP to certificate: 127.0.0.1 09:38:08.660 starting Users/k/.local/bin/rclone-1_60_1 09:38:09.418 detected webdav address: https://127.0.0.1:53263/ 09:38:09.582 throttling limits from connection info {"limits":{}} 09:38:09.586 finished initial cache scan {"cache":"contents","duration":"2.787934ms","totalRetainedSize":60,"tooRecentBytes":0,"tooRecentCount":0,"maxSizeBytes":5242880000,"inUsePercent":0} 09:38:09.649 finished initial cache scan {"cache":"metadata","duration":"62.684891ms","totalRetainedSize":95279724,"tooRecentBytes":71033663,"tooRecentCount":61,"maxSizeBytes":5242880000,"inUsePercent":1} 09:38:09.667 finished initial cache scan {"cache":"index-blobs","duration":"17.742097ms","totalRetainedSize":315541878,"tooRecentBytes":151342378,"tooRecentCount":55,"maxSizeBytes":5242880000,"inUsePercent":6} 09:38:09.668 refreshing committed state because it's no longer valid (now 2023-05-01 09:38:09.668047 +0200 CEST, valid until 0001-01-01T00:00:00Z) 09:38:09.668 refreshAttemptLocked 09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":1} 09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":2} 09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":4} 09:38:09.668 [STORAGE] concurrency level reached {"maxConcurrency":3} 09:38:13.661 [STORAGE] ListBlobs {"prefix":"xw","resultCount":1,"error":null,"duration":"3.993067599s"} 09:38:13.662 [STORAGE] ListBlobs {"prefix":"xe","resultCount":2,"error":null,"duration":"3.99385125s"} 09:38:16.559 [STORAGE] ListBlobs {"prefix":"xr","resultCount":5,"error":null,"duration":"6.891036144s"} 09:38:16.560 ranges: [{xr8_15_8bb3bf1ccc5e2369410e183ca75e812a-sa449d0540eacac8c-c1 10420663 2023-02-20 07:13:50 +0000 GMT} {xr0_7_9a9d2b0fb54b712a13d3b3681655c91e-sc802fceef3ed2710-c1 33320996 2022-12-14 02:14:20 +0000 GMT} {xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 60384988 2023-04-30 18:30:30 +0000 GMT} {xr0_21_c2d51b5cadac51c70a6248cbe3f3118f-s8acd5e95ac0ba410-c1 51911353 2023-04-09 00:08:12 +0000 GMT} {xr0_20_d4100dd696859736ab06aa4cecac0477-s26696249ab7724dd-c1 50161711 2023-04-04 13:42:52 +0000 GMT}] 09:38:28.244 [STORAGE] ListBlobs {"prefix":"xs","resultCount":30,"error":null,"duration":"18.576212592s"} 09:38:29.057 [STORAGE] ListBlobs {"prefix":"xn27_","resultCount":0,"error":null,"duration":"811.70718ms"} 09:38:31.988 [STORAGE] ListBlobs {"prefix":"xn26_","resultCount":10,"error":null,"duration":"3.742948669s"} 09:38:33.831 [STORAGE] ListBlobs {"prefix":"xn25_","resultCount":20,"error":null,"duration":"5.586130433s"} 09:38:33.831 current epoch 26, uncompacted epoch sets 20 10 0, valid until 2023-05-01T09:58:09.668173+02:00 09:38:33.831 not generating range checkpoint 09:38:33.831 adding incremental state for epochs 25..27 on top of [{xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 60384988 2023-04-30 18:30:30 +0000 GMT}] 09:38:33.831 Complete Index Set for [0..27]: [xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1 xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1 xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1 xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1 xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1 xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1 xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1 xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1 xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1 xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1 xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1 xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1 xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1 xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1 xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1 xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1 xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1 xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1 xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1 xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1 xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1 xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1 xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1 xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1 xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1 xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1 xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1 xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1 xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1 xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1], deletion watermark 2023-02-06 22:56:31 +0100 CET 09:38:33.831 active indexes [xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1 xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1 xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1 xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1 xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1 xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1 xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1 xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1 xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1 xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1 xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1 xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1 xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1 xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1 xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1 xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1 xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1 xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1 xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1 xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1 xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1 xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1 xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1 xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1 xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1 xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1 xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1 xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1 xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1 xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1] deletion watermark 2023-02-06 22:56:31 +0100 CET 09:38:33.832 use-indexes [xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1 xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1 xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1 xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1 xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1 xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1 xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1 xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1 xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1 xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1 xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1 xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1 xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1 xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1 xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1 xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1 xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1 xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1 xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1 xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1 xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1 xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1 xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1 xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1 xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1 xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1 xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1 xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1 xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1 xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1 xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1] 09:38:33.833 combined index segments {"original":31,"merged":28} 09:38:33.833 expireUnused {"except":["xr0_24_ac43b08886332a76682030ba940fdac6-s8f43dbc8699b4f3f-c1","xn25_f217a33bcc16fe47f2ab8919dfd0eee4-s4f6a6226134226ad11b-c1","xn25_e829669100cce1bcebc6804bd7b0d8b9-s3246768efc33834711b-c1","xn25_ea5decfe0d1aa69b30216888065126df-s97d539a3c027c0dd11b-c1","xn25_d6253a50cca75134df0621e493150b34-s768413a27f4c965611b-c1","xn25_da45d5bd49cf4e1da0fe5845d1023c1a-sea2ebb62c9550a7411b-c1","xn25_ca5ec636dac15100a827f59483c52c13-se3c1c69d6f1ab8b911b-c1","xn25_cc0b96a11327421d502d2151f7c4c9ee-s3d35b4d5ed1cb25011b-c1","xn25_b9794e93030e260cfd21d01346541d78-sc4e84e30720a8fd111b-c1","xn25_94079ddbb31f285a6d60bff408799288-s93217da23ae27af111b-c1","xn25_9d32079596a17c02eb6a2aa75339c730-sc095858b28cfefee11b-c1","xn25_7268f66979ed7247352ada1514690e73-s5ca48d6ae280e44b11b-c1","xn25_52b1505695d5d3a0d976a8702448abd2-s6329a4d0f1c9fc2a11b-c1","xn25_27d722c2f5f58b4741c300fe45c6a2ad-sbe996b501e512ebb11b-c1","xn25_294557d072303b529f2953b87efe398b-sc54acb70587e356311b-c1","xn25_2e92c8a300c972d2745b2bfa8feab9d3-sc13043274e30399d11b-c1","xn25_135cf288ff986b25da9a2689a2b478de-sea0435a3a984b4ea11b-c1","xn25_19929ecb1b1168e08a561a666895b983-s61210645d5490c6211b-c1","xn25_0cab55f22eb8d889841b7e97a681626d-sa0ef62edc324190011b-c1","xn25_0cda54540b2b4f6b2401f696fd5c5aa0-s1aca91b59bda281511b-c1","xn25_0f8135d9f6b1e4cc9c5a90e282705c15-s82858b21a1b18c2711b-c1","xn26_e15c92628dc26584e360bac905bf679d-s8b11201f1131f20011b-c1","xn26_e67d7082e596af34c2870b1e8dcc679c-sd47a13c165bbcd6611b-c1","xn26_ef41e970910f1744f5ad53bcf0466cad-s49d8a3fccd6125c511b-c1","xn26_d44dcaee3a230e9d1b96964cfe7941f0-s943acfb04bfb842211b-c1","xn26_d7400c02a8fef40ee2606153cb2b3f2b-s6c92cef71b93393a11b-c1","xn26_c696bf8e09461b1fab5e73ef3cc65238-sd1bb406c73358e3411c-c1","xn26_bb62240e0fafadd9840da64727399e03-s31df0c18e8ccebbb11b-c1","xn26_8752699ecd2f1046957c94867a3ef42d-s3938dd91d571515111c-c1","xn26_6fbe3a5fe14a7f919bb9c3e9dc771d69-sfa87237ddcc6738a11b-c1","xn26_0e82a268cd608b2186277c2401190823-sca5e7ea2449ad62a11b-c1"],"minSweepAge":"1h0m0s"} 09:38:36.662 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"2.828023695s"} 09:38:36.662 getContentInfoReadLocked(m079d5af4425d88f0a5a62c45c0d4c415) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found 09:38:36.662 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"482.169µs"} 09:38:36.662 getContentInfoReadLocked(m3a8db689ff9d228135809c6d903999ad) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found 09:38:36.663 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"396.692µs"} 09:38:36.663 getContentInfoReadLocked(m47793d0d73bddc091df13f9f2610b1a1) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found 09:38:36.663 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"446.51µs"} 09:38:36.663 getContentInfoReadLocked(ma9934163cc21c036a0fbfbbe8ff636ae) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found 09:38:36.664 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"400.725µs"} 09:38:36.664 getContentInfoReadLocked(ma7d74bec69eb8e385dc13d9cbbf415fe) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found 09:38:36.664 [STORAGE] GetBlob {"blobID":"q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117","offset":0,"length":-1,"outputLength":0,"error":"BLOB not found","duration":"395.983µs"} 09:38:36.664 getContentInfoReadLocked(me986cf021a6b3304548993a62e500273) error error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found ``` </details> Then (probably after restarting KopiaUI), the repository doesn't show up in KopiaUI at all, it just shows the repository selection screen. If I try to reconnect to the repo, it goes to the `Initializing repository...` screen, then afterwards it just resets to the repo selection screen again without any error message. <img width="350" alt="image" src="https://user-images.githubusercontent.com/11315492/236018925-8a78d4c6-92b8-4ec3-acd8-5ea7e213a0a3.png"> Even though that screen shows up, I now see that I do have a `repository.config` file: <details> <summary>repository.config</summary> ```json { "storage": { "type": "rclone", "config": { "remotePath": "kbackup:kopia", "rcloneExe": "/Users/k/.local/bin/rclone-1_60_1", "atomicWrites": false, "dirShards": null } }, "caching": { "cacheDirectory": "../../Caches/kopia/b9130b74a7154aba", "maxCacheSize": 5242880000, "maxMetadataCacheSize": 5242880000, "maxListCacheDuration": 30 }, "hostname": "k-imac", "username": "k", "description": "My Repository", "enableActions": false, "formatBlobCacheDuration": 900000000000 } ``` </details> If I run `/Applications/KopiaUI.app/Contents/Resources/server/kopia policy list`, it gives me a similar error to the one I originally got: > ERROR error listing policies: unable to list manifests: unable to load manifest contents: error loading manifest content: error getting cached content: failed to get blob with ID q45ab4d9c3fc42a7b455aea68f6a4d5a6-s633b8c13200b13ba117: BLOB not found

Is there a way that I could recover the repo (I don’t mind losing snapshots, but I don’t want to setup the policies every time this thing happns)?
Should I just open Kopioa less regularly so these things don’t happen? I smell that NTFS is not the most stable file system and that could affect, but IMHO Kopia should help us on recovering from this things.

Topic		Replies	Views
Can not access repository, failed to get blob Support	15	2155	July 18, 2022
New Client failed to initialize: error getting cached content, blob not found Support	1	332	March 9, 2023
Problem connection to repository Support	10	649	August 31, 2021
BLOB missing: kopia won't work Support	3	533	February 18, 2021
Kopia Deleting blobs prematurely Support	0	242	November 16, 2022

Can't open repo, manifest corrupted and BLOB not found repo

Related Topics