Unable to perfom maintenance to reduce repo size

Support
1

My Kopia repository is eating up my disk space and I find no way to reduce it’s size probably because the maintenance run fails due to “unable to rewrite content” errors.

My recent snapshots hava a total size of about 320 GB and it is mostly data does not change a lot so the previous snapshots do not add a lot of space. But my repo is almost 900 GB in size.

Maintenance shows 430 GB of unused contents but I am unable to get rid of these unused contents.

The problem seems to be that due to missing blobs the maintenance fails and does nothing.

Output of “kopia maintenance run --full”

Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
... found 100000 unused contents so far (430.3 GB bytes)
GC found 113378 unused contents (451.7 GB)
GC found 0 unused contents that are too recent to delete (0 B)
GC found 146469 in-use contents (314.1 GB)
GC found 612 in-use system-contents (675.6 KB)
Rewriting contents from short packs...
unable to rewrite content "61e6a8524662f3e5017ecf14b10d9864": unable to get content data and info: error getting cached content from blob "p42ef5543bdf349011427d2312bd1f29d-s92ab867c3cf00bff11c": failed to get blob with ID p42ef5543bdf349011427d2312bd1f29d-s92ab867c3cf00bff11c: BLOB not found
unable to rewrite content "6dc61845de44f0fcf1d1dcdfbee15c94": unable to get content data and info: error getting cached content from blob "pacd5551805b0fec82c9ae01c1d55c0a8-s342b32f1460c4f4e11c": failed to get blob with ID pacd5551805b0fec82c9ae01c1d55c0a8-s342b32f1460c4f4e11c: BLOB not found
unable to rewrite content "a53e61f23a2bcfdf392fd398132222d2": unable to get content data and info: error getting cached content from blob "pacd5551805b0fec82c9ae01c1d55c0a8-s342b32f1460c4f4e11c": failed to get blob with ID pacd5551805b0fec82c9ae01c1d55c0a8-s342b32f1460c4f4e11c: BLOB not found
unable to rewrite content "2da1f9530612accfb9f479f94a80330b": unable to get content data and info: error getting cached content from blob "p40e97afc4a75e10f5329e1afcdc82c6b-s8ca6410043b375df11d": failed to get blob with ID p40e97afc4a75e10f5329e1afcdc82c6b-s8ca6410043b375df11d: BLOB not found
…
Total bytes rewritten 1.1 GB
Finished full maintenance.
ERROR error rewriting contents in short packs: failed to rewrite 50 contents

I get exactly the same result on subsequent runs.

What I tried so far

kopia content verify
==> No errors

kopia blob gc --advanced-commands=enabled --delete=yes
==> Removed 110 GB of blobs but the maintenance error persists

kopia cache clear
==> ERROR error removing cache directory: unable to complete delete cache despite 10 retries: unlinkat /app/cache: device or resource busy

kopia snapshot fix invalid-files
==> No changes on any of my snaphots

In my despair I also run kopia content delete -advanced-commands=enabled
==> Still no change

So I am out of ideas. Any suggestions how to solve this?

Regards
Bernd

2

and given that you see error:

I would think that there is some problem with your local cache. Delete all cache folder manually (rm -rf /path/to/cache or whatever it is on your OS). It is 100% safe as its content is ephemeral anyway.

3

Thank for the suggestion. Unfortunately it did not solve the problem.

Stopped my Kopia Docker Container, deleted the cache folder, restarted the container and rerun maintenance. Got exactly 50 content rewrite errors again.

4

Try running maintenance from another machine. Maybe without docker to keep things simple. These error messages might be misleading often but given that kopia struggles to operate on its own cache it sort of points into some local issue (not enough space? permissions?).

5

Sorry, same result.

Stopped Docker, connected from another machine and rerun maintenance.

6

kopia cache clear is probably safer :smiley:

Could this be caused by file permission issues? Can you run kopia repo validate-provider to see if there are general issues with your repository?

7

I did remove the cach whil docker was down, so probaby no issues whil deleteing the cache folder.

I did run “kopia repo validate-provider”, Resul ist “… All good”

What I do not understand ist that I cannot just delete the listed content blobs while accepting that some data is not recoverable then. The alternative seems to start from scratch so ALL historic backup data would be lost.

8

I gave up… :frowning:

I set up a new repo for new snapshots and kept the old repo for (at least partially) possible restoration of data.

I hope I do not run into this issue again…

There is one thing I could imagine causing this issue: Kopia runs in a docker container (of an Unraid server) for simplified maintenance and updates. However, I did connect to the same repository using SFTP from a windows machine. Maybe there could be an issue with concurrent access or the windows machine alsy trying to do do a snaphot at the same time the docker container is doing one… :thinking:

Anyway, thank for the suggestions to fix this.