I’m running nightly backups of 2 btrfs partitions using systemd. I use btrfs snapshots so that there are no modifications to data during backup.
So basically I have 2 backup jobs which run at 4:00 (I assume it’s not a problem for 2 backup jobs for different disks to run in parallel?) and a kopia snapshot verify --verify-files-percent=1 --file-parallelism=10 --parallel=10 which runs at 5:00.
Today the verify job failed with some errors:
Processed 204606 objects.
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/usb_modeswitch/12d1:155a: error reading object 3bdca67c4a78d523579b77bcadee2109: unable to open object 3bdca67c4a78d523579b77bcadee2109: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,396352,110) despite 10 retries: java.net.SocketTimeoutException:5 SECONDS
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/usb_modeswitch/19d2:1020: error reading object 2eb8cd28c08098d7479ae1f99d596438: unable to open object 2eb8cd28c08098d7479ae1f99d596438: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,409434,114) despite 10 retries: no shard headers could be fetched
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/usb_modeswitch/12d1:14c5: error reading object 669786bd250e079b70bf8de9ac162ead: unable to open object 669786bd250e079b70bf8de9ac162ead: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,394980,111) despite 10 retries: java.net.SocketTimeoutException:5 SECONDS
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/setroubleshoot/plugins/bind_ports.py: error reading object c3bede4392c757c509792be7a4a487e5: unable to open object c3bede4392c757c509792be7a4a487e5: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,625320,1399) despite 10 retries: file_state_unknown
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/mime/text/x-authors.xml: error reading object 6524294dd0365c7daecc0af7d165d4e5: unable to open object 6524294dd0365c7daecc0af7d165d4e5: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,15040,1055) despite 10 retries: file_state_unknown
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/usb_modeswitch/1076:7f40: error reading object 5adc522ace0dead458a1ac6a35fb75cd: unable to open object 5adc522ace0dead458a1ac6a35fb75cd: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,391384,104) despite 10 retries: file_state_unknown
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/usb_modeswitch/19d2:ffde: error reading object 21153f0aef191adbabd5a5a700af6b50: unable to open object 21153f0aef191adbabd5a5a700af6b50: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,414061,123) despite 10 retries: no shard headers could be fetched
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/setroubleshoot/plugins/samba_share.py: error reading object 0e35375e2913ebde1e06f3719a8f78ed: unable to open object 0e35375e2913ebde1e06f3719a8f78ed: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,662903,1417) despite 10 retries: java.net.SocketTimeoutException:5 SECONDS
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/usb_modeswitch/19d2:0003: error reading object 515f57933c6b4a79490cd350dd18aaa3: unable to open object 515f57933c6b4a79490cd350dd18aaa3: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,405931,73) despite 10 retries: no shard headers could be fetched
ERROR error processing root@lkserver:/@2024-05-30 13:04:38 CEST/usr/share/usb_modeswitch/2001:a403: error reading object cf311b2510aa7df07dd5c5025900dcd5: unable to open object cf311b2510aa7df07dd5c5025900dcd5: unexpected content error: error getting cached content from blob "p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129": failed to get blob with ID p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129: unable to complete GetBlob(p8cf1f97b739f157f057e63c617f65317-s887f5a45be7377d3129,423403,126) despite 10 retries: file_state_unknown
Finished processing 207557 objects.
ERROR encountered 10 errors
What I’ve tried so far:
- Running verify manually to make sure it’s not a 1-time error
kopia maintenance run --fullsudo kopia snapshot fix invalid-filesand latersudo kopia snapshot fix invalid-files --commitsudo kopia repository repair s3 <flags>- it just saidformat blob already exists, not recovering, pass --recover-format=yes. When I passed this flag it went trough many blobs and at the and saidERROR could not find a replica of a format blob
How can I fix those errors? Why did this happen and what should I do to avoid this situation in the future?
Out of curiosity: why do I see java.net.SocketTimeoutException in the logs? I thought that Kopia is written in Go?