Maintenance Failed due to missing blobs, content verify fails, next steps?

Hi, it’s me again, I swear I’m not trying to be a bother, lol.

Looking in the UI I noticed that maintenance was failing, so I investigated a bit. Here are some relevant log lines:

19:58:16.722	unable to rewrite content "kf8a91c2a77e72427a9ddad647c05deaf": getCacheForContentID: BLOB not found
19:58:16.757	unable to rewrite content "kfa79f010eec4fca33d26c52167c3eabb": getCacheForContentID: BLOB not found
19:58:16.790	unable to rewrite content "kfbd8ae86db5e296a6f8ae7c32a039f25": getCacheForContentID: BLOB not found
19:58:16.827	unable to rewrite content "kfc1a175e4056a618aa963592ea97c080": getCacheForContentID: BLOB not found
19:58:16.860	unable to rewrite content "kfec3cadc0574b78871b9106aa08003c1": getCacheForContentID: BLOB not found
19:58:16.895	unable to rewrite content "m1c2812cf4e33eb354dd88662b1207c5f": getCacheForContentID: BLOB not found
19:58:16.930	unable to rewrite content "x1db6c5f0f72081b8b32e50ea8173c389": getCacheForContentID: BLOB not found
19:58:16.971	unable to rewrite content "x1ef57a57a5bf4cdb115f43d3c1a84434": getCacheForContentID: BLOB not found
19:58:17.014	unable to rewrite content "x420cfb56984e2d72aec7b1a74947cfb4": getCacheForContentID: BLOB not found
19:58:17.050	unable to rewrite content "x5808c95fe1b69ad48d172bacd15a2a3e": getCacheForContentID: BLOB not found
19:58:17.089	unable to rewrite content "x5f15cfd53fb8d2965b194ce029509dc0": getCacheForContentID: BLOB not found
19:58:17.124	unable to rewrite content "xa5452f2864fcaeb62c5dbe6e14dee196": getCacheForContentID: BLOB not found
19:58:17.159	unable to rewrite content "xb31053ca442c317b73d09822fda78699": getCacheForContentID: BLOB not found
19:58:17.199	unable to rewrite content "xd154780ddd4f10fc26ddb132ec1d8847": getCacheForContentID: BLOB not found
19:58:17.243	unable to rewrite content "xf45c7165437164c5ab40ecc5aedce249": getCacheForContentID: BLOB not found
19:58:17.243	Total bytes rewritten 192.5 MB
19:58:17.426	Finished quick maintenance.
19:58:17.426	not closing shared manager, remaining = 1

after seeing this, I ran a kopia content verify and got this:

❯ kopia content verify
Listing blobs...
  10000 blobs...
  20000 blobs...
  30000 blobs...
  40000 blobs...
  50000 blobs...
  60000 blobs...
  70000 blobs...
  80000 blobs...
  90000 blobs...
  100000 blobs...
Listed 108013 blobs.
Verifying all contents...
  100000 contents, 0 errors...
  200000 contents, 0 errors...
  300000 contents, 0 errors...
  400000 contents, 0 errors...
  500000 contents, 0 errors...
  600000 contents, 0 errors...
  700000 contents, 0 errors...
  800000 contents, 0 errors...
  900000 contents, 0 errors...
  1000000 contents, 0 errors...
  1100000 contents, 0 errors...
  1200000 contents, 0 errors...
  1300000 contents, 0 errors...
  1400000 contents, 0 errors...
  1500000 contents, 0 errors...
  1600000 contents, 0 errors...
  1700000 contents, 0 errors...
error content k00757b6ca3ebac41e01ab982420806fd depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k00eb39a9309637a814cdf8d7acb8c456 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k01b6d56bd890808b9f99a2514497c4be depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k022c72424f0a830616c4936d48c7d703 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k027ede4d565d0fef151eb6d2d2a20111 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k031de14d3264511e523d87fcb08a9869 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k04ebb187e09b2e8007b97aeac9295d26 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k071bad2b7357e3dcba3f5af8b859ba37 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0780a1cd4e2a0c1ba3eea893265e8efc depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k07f3611b96a12ac4e243248b72ac3d94 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0948a2b776c897ef90ddbc7f42e114dc depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0afea13a7c116559b3ec823a82d99bdf depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0c3ad16ebf0c447106d87ede5b87608e depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0cabe0d72a3d6f7fca2cbb47bab9725c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0cea764e9a9253d3e458f8d897545a1a depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0d27ae00a29bff087f2477f970c8513d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k0ee794b004e1179f6ff828379d07cb7d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k115f3376ed75badf3af6d6d66ba3ca6f depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k1657e8589607dc9aba36df13df0d3655 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k16ad7373aadf25fcd37adf082147e33c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k17fe912cd7c55166f16e509c539668c9 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k1bb2d191b4f4deba54b94800dfc5abc6 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k1e40f03aa5b9b00aabb0927dcb439c46 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k1efa3756a8f9bc93e1a54783e1a111a2 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k1efcc44c6adcb6a47f8b51c8c495371c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k1fca7cad030de5b2bdc05dd9ff3c3c37 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k21ca173e3dd20f8773850235969401fd depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k21e9ea4549e9b97f3f85f7361eafbf4d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k250088b970745de71f005335bc79b8d4 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k25cfe94d6e3265c35ce4aca9cbe38115 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k263137a0af9cf0e21ea3bb4c1b4a364a depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k2961eef69a02272b554a558ade2a0c73 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k2a08d306f0e438ca78c6a0d1cf5ca258 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k2a44e934dd4a27bf97881dfaa5229d42 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k2c78390498adb005f86b6d112f99c85d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
  1800000 contents, 35 errors...
error content k2fcdcc3e25607bbf3ee819b4bd367ac8 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k3294e4831e4b9b768eee56d56f8c5efd depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k3371e91984b42c87136ad3d158b1eeec depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k33c022836140dc9af404d71e9f9aaefe depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k354c8e30f3f88d249e0e7496730dcf10 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k36119236b17cc4c76d40098743513123 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k3755e48c5a2639c03cb33daab49db8f1 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k39b40de31f21913206a9f580d3a20808 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k3bb9fe3626880aeb050a19abde0278b9 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k3e4cb61b82e39497e8e8f9cbcb16179d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k3f24c29912ea5cbd8dac835c990d3977 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4045e1b3f00da2614e230cb95e8610e2 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k40d14a06a7d223012813d61cba0ce0e4 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k40e87734cb7c2f2a4d691d3d499b8f37 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k442342629698784bfb0396812f8dd9bf depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4506331045b68c3ac279e29a2cde61fb depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4949543b4f82aa7cc3fa60f2ada0e89f depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4aacfdcadaa11beb601a0f5819f43d07 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4b544308cf3ea38d816760aded903c33 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4b986c00735bfbf47e0feafce6d3cee7 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4bb57bdbeca4ec07bbde6fa1c7ea14dd depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4bbe9be2085ee7c835ab2ddc8c779cdc depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4c50020696736432b86ddbf4750a9f27 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4d900f944748db2552e91f4d057ef4cc depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4dfa826e955ae5e8cf487e3feb26433c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4e254046a811dbeb45a40dab20e7710a depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4e37e55c3196b0ec366553690fbd0822 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k4fca1321535f0189f60bd96b4d88fa67 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k50e173ec9e2008b2def2aacb946bd16c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k519a2025ac69ea1ea14467474b28864c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k5750bfa2df9d05124248463cf48c0aab depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k59d05818201339736158437b80f1c7af depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k5b1df047ce1468ff5e80dc21c0dfd56e depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k5cbd66feb5acbfb1d150676409fb3480 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k5f68606b747727ed798379ab882665b4 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k5fcea29a015350d11ce9ee21d932d2a9 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k60f4154b295c6e63cb9bda66b27a32b2 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k61e3023c686cb42a60c6e57b5bb8c20d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k64c34949c40f6c9303b441096fa4b1e7 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k64ef7327271587c17763d800771d5102 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k65211195b3a1f7b88c3668c49c94cc45 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k66da0299a289ecdc1288ccc88bde7914 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k68083a96a9172e578786ff9f3ed1cdb1 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k68423efde73ffcd495416270d9f1cd3b depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k69be63264feaa2cfaaa64056176cd7fe depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k6b35c54b089d676981406fd221ae6380 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k6bc2dd4d86b3657b58ea63c56cd36000 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k6c8da236def8b20ae6357c8c818e66ae depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k6d5790523a55c79947e99d4ccc9ee81f depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k6ed799bafbf229bd73c45ed7b4c7239b depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k6f7328bd18ccd772bbcf77e6b31ad9b9 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k7322d2d978da7e95d8390ff439d939a0 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k74ed75aa2b5e7784b4acd1a92581cd0e depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k76a389f9f5b4c7caddbbcfc2e0579fbc depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k79c8c33f5e0a3c5fa10748f0af7ef911 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k7a3047daaadc2a6856006e566e407290 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k7c4aeed3967b6c8f2afc77a4c20f1707 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k7dfa43f3c12ef47affc362722d592d73 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k7dffa414d84ed9a61dc7da074693248d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k7faa264a89e1d028f26bef8b5b808efc depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k81403704f66f48a776c7b7181512d52b depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k82bc1644af49c6330e9314e9cc971822 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k83eb19a97f3eec2741c06b3fab1c3da0 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k85ea0de806f3ab31dcc1babf4247c7cd depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k862d2ee81393ea7325f290b87003a5f4 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k88d447cca9b8cbc09f3ac1b802cad66b depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k8a17f267738381735acb729caf95d609 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k8a224089bd8fc26222ad884ac1dc2e44 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k8a8276eb82b47705d0552b7ad3a68ed7 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k8b2fa9b5969c5bfb2612fa56bb219159 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k90fac791bd80c3e1bc1c22b824a9c5ae depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k9189256d14bda0ee92e633062884f6d2 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k942119ba1c234736061a441f037ae6b1 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k947113c32feecb8917e3efe108d4d67c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k9aa7cb9a5ffa8765bda0fa60e9760723 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k9b084ea849eb3d2248b482ff5c8fc204 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k9c9e710b3e39dc134ee3b451a2267a4b depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k9d953cd3b3a7f94e1f0b7d06dfe2198e depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content k9e0f88f6a668c11afd8de8a18462d165 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ka0812151c3488f5005f6ad0237539eba depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ka0ab93cc05490af368fa326dc3348462 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ka2f5a3938423fae950fbf24c2af3cd7a depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ka42750f8bfbfb77a2c96150aa1b3f9c3 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ka82344b45c86c5532bab4bb2a9650c65 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kaad8d628f2b30db33e9539656c84a9ff depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kac285d0e8c3608419dfc015b255d7881 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kac926f8cb3b48c68a9fcac25826535e3 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kaf83b2ddede6c8a7fcd991a2203aaeff depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
  1900000 contents, 123 errors...
error content kb18cc6a30d2144a98207440387e08b59 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kb1c2a75598ae15d16178cd1918eb194b depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kb24c65f9a8aa9be999bec5826d5bb649 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kb77c5d6fdf9838084b9d9304f3dcce69 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kbddda1546d97d0381b41052622478eac depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kbf1a75ffaf6b93eddacd0f4c15c30e71 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kbf363d4df27d1ab15641af37b246ebee depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kbf7f8aeeebcc41773e04c57301430348 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kc02a124f2301a4d6fac69a7eb7478fae depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kc24553702a6ee6c468500e955117cfab depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kc26690f6349ffc610aafe8046683662d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kc324ff270679d3e57ccfc1cc4ff1bf4a depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kc4704a2b8dea03791b6da79135694a44 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kc5d6088d5ed05ec9f670b28281277e3a depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kc8fe8704d9c9cdfa3a1c882ea0b213af depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kcb89213762d4318268e2f85930f8969c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kcdcd68a0a54dc32f5233e1cc470581bf depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kcf1d18225cedcb27eda9215e8352a75e depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kd43f68662b7c6b5abdc680bb7d8e5e0d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kd6a274d6761dfae2a1f1898cb9c49cb4 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kd8df93bd9701391b4ba792f02754e130 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kd9b2631674f14ec55c5584698f5f6c2d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kda0de44c19746cfe13cd2bce1215940f depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kdad6dcfbbd71def08f988a81328f9a9c depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kdc726dc73aaad2008d9ddde633d229ed depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kdd6b96bdd5007938a4799ffaa8081008 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ke364ba8100eeb650901586bd312b639e depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ke5c71f5c9e51b0e0e76e561040beee19 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ke5eddda29db98daa1536bef7a52c3b94 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ke7b1d520436d9c869d184af2b025c6de depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ke8d8fde84fb9f4ce8660b07bdd49699d depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content ke8e8d26f77bfa1f3a660f801ec56c6b6 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content keb54cb1db431eeb5ad2477da0f8d7632 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kec1627d0a619102c0516c2bda0ce3f10 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kedc9b43e9ee18151a4f823571ec5c879 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf348280fe270e802f493b298febf6891 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf374539ecd87a34e33027e7056e7695f depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf40d79b927b5675957f43ffeaa8b20a3 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf5587ff1f045bdaaeef0038f765f95f6 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf748105bba2cc00da75ae5a98d0f72c0 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf80459e1f31b24d993ce6b889a4f15fc depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf81d545814f9fac7020f54e681854e26 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kf8a91c2a77e72427a9ddad647c05deaf depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kfa79f010eec4fca33d26c52167c3eabb depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kfbd8ae86db5e296a6f8ae7c32a039f25 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kfc1a175e4056a618aa963592ea97c080 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content kfec3cadc0574b78871b9106aa08003c1 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content x1db6c5f0f72081b8b32e50ea8173c389 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content x1ef57a57a5bf4cdb115f43d3c1a84434 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content x420cfb56984e2d72aec7b1a74947cfb4 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content x5808c95fe1b69ad48d172bacd15a2a3e depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content x5f15cfd53fb8d2965b194ce029509dc0 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
  2000000 contents, 175 errors...
error content xa5452f2864fcaeb62c5dbe6e14dee196 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content xb31053ca442c317b73d09822fda78699 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content xd154780ddd4f10fc26ddb132ec1d8847 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
error content xf45c7165437164c5ab40ecc5aedce249 depends on missing blob q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106
Finished verifying 2039227 contents, found 179 errors.
ERROR: encountered 179 errors

The only thing I have changed recently is moving full-maintenance to every 10 days. What are my next steps to try and resolve this? Thanks so much for any help.

Looks like one blob went missing. Can you ping me on Slack (slack.kopia.io) so we can debug interactively. I would first grep logs for mentions of q7e08993523722d521570dc6e9b2f295e-s47ca2d58c19e9ae4106 and see if there is any indication of why it is not there.

1 Like

Can do. I’m at work until about 12 hours from now, so I’ll join the slack then and if needed we can work on this asynchronously. I’ll look through logs around the same time.

Is there any harm in letting my clients continue to snapshot or should I stop anything new from being added till this is resolved?

I would let clients continue.

1 Like

for anyone that runs into something similar with a q-blob, and just so i can mark this post as solved, you should:

  1. package up all your logs from all your clients
  2. reach out to Jarek on slack
  3. check each client’s local cache under metadata/<first 2 characters of missing q-blob>/<remaining part of blob>.f
  4. if you find it, copy that file out to another location, rename it to match the missing q-blob, and upload it to your backend.
  5. kopia maintenance run
  6. kopia contents verify

really appreciate all of Jarek’s help during this.

2 Likes

To be fair. The root cause is still under investigation. Something caused the premature deletion that we need to find and fix.

1 Like