I have two host machines which backup to the same Kopia repository connected through rclone. The rclone remote is Onedrive. I’ve had no issues with backups or maintenance tasks for the past several months.
Recently, I mounted a new (large) drive to one of the machines and forgot to add it to the ignore list so I ended up with a massive backup, whoops. I updated the backup policy to exclude that mount folder and deleted the large backups but noticed the space on Onedrive was not freed up despite doing so. I forced a full maintenance and saw that I’m getting errors during this:
coanghel@server-01:~$ docker exec -it kopia kopia maintenance run --full --safety=none
Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
... found 100000 unused contents so far (260.4 GB bytes)
... found 200000 unused contents so far (521.1 GB bytes)
... found 300000 unused contents so far (781.6 GB bytes)
... found 400000 unused contents so far (1 TB bytes)
... found 500000 unused contents so far (1.3 TB bytes)
... found 600000 unused contents so far (1.6 TB bytes)
... found 700000 unused contents so far (1.8 TB bytes)
... found 800000 unused contents so far (1.9 TB bytes)
... found 900000 unused contents so far (1.9 TB bytes)
... found 1000000 unused contents so far (1.9 TB bytes)
... found 1100000 unused contents so far (1.9 TB bytes)
... found 1200000 unused contents so far (1.9 TB bytes)
GC found 1255325 unused contents (1.9 TB)
GC found 0 unused contents that are too recent to delete (0 B)
GC found 1144224 in-use contents (49.9 GB)
GC found 395 in-use system-contents (6.4 MB)
Rewriting contents from short packs...
unable to rewrite content "3ce4ea57e66ef8489d7fe043a555cbde": unable to get content data and info: error getting cached content from blob "p68906bcd60baa8e97bf11f9144f604d2-s0b652585bad6c687125": failed to get blob with ID p68906bcd60baa8e97bf11f9144f604d2-s0b652585bad6c687125: BLOB not found
...
Total bytes rewritten 78.7 MB
Finished full maintenance.
ERROR error rewriting contents in short packs: failed to rewrite 2647 contents
I’ve tried to force a blob gc, manually delete the missing blobs, and reset the index from the host which owns the maintenance:
kopia blob delete --advanced-commands=enabled <the blob>
kopia blob gc --advanced-commands=enabled --delete=yes
kopia index recover --advanced-commands=enabled --commit
I’ve also tried to fix invalid files from both hosts:
kopia snapshot fix invalid-files --verify-files-percent=100 --commit
Anything else I’m missing? Honestly at this point I’m pretty close to just deleting the repo entirely and starting from scratch… but I’d rather not abandon several months of backups (and also would like to know the right way to recover from this in the future)