My backup is blocked because there is not space left on the device I’m backing up to. This happened because the current retention policies are not aggressive enough.
I have already updated the retention policies, but since the storage device is full, I cannot delete some old snapshots and run the maintenance to really free some space (the maintenance process requires some free space for temporary files).
Which is the best way to free some space?
Is it possible to identify a given file or directory in the encrypted kopia files in the backup? So that I would directly delete them?
Another alternative would be to mount the backup on a local machine and delete some files from the mounted backup? Is this possible without ruining the rest of the backup?
Good luck with dealing with it. There is plenty of posts in this forum about this subject.
It is known “problem”:
but without proper solution yet. If it was me I would just start from the scratch as it seems that there is no clean way to recover such repo “inplace”. Which also means that you can always copy/move all kopia files somewhere else (with more free space) and do all required maintenance there.
Losing one repository should not be an issue if you follow some basic backup strategy.
In the future I recommend always to have some space holder file in your repo destination. I always put 1GB one just in case I run into issues like you now. It is simple and reliable way to have a safety valve:) For kopia and any other software.
Monit is a AGPL 3.0 licensed process supervision monitoring agent for Unix/Linux. It has the ability to preform user-defined actions based on the state of its host. The default polling ‘cycle’ is 30 seconds which is adjustable.
It has a space usage test, eg:
/etc/monit.d/stor-sdb1.conf
check filesystem storfs with path /mnt/stor
every 2 cycles # poll every 1 min
group storage
group stor
start program = "/bin/mount -a /dev/sdb1 /mnt/stor"
stop program = "/bin/umount /mnt/stor"
onreboot laststate
mode active
if does not exist then start
if space usage > 90% then alert
if space usage > 99% then stop
Other actions are restart, exec and unmonitor based on percentage, B, KB, MB and GB. This allows the ability to go as far as unmounting the storage device (as shown) if not simply terminating individual or grouped processes (eg: Kopia-UI, Kopia Respository Server, WebDAV related daemons, etc.).
It is coded in C which I understand Golang draws certain similarities.
Surely Kopia devs can adapt an approach and incorporate it accordingly.
Kopia is open source project so if you feel like you need something badly and have skills to implement it then submit PR. Otherwise you have to wait (maybe forever) until somebody else does it.
If you look at kopia github you will see that there is not so much new features development happening nowadays so realistically without some community member working on it I doubt it is going to happen.
I already have disk monitoring in place, per above. Others do not.
In the interim it might be advisable if someone updated the docs showcasing this workaround/solution for, as you state, this “known problem” with numerous posts given Kopia dev’s oversight to incorporate quotas. I see they’re not against incorporating nginx conf snippets.