Hi! I’m using kopia now for a couple of month now and it works really well. Normally, I start the Kopia UI at startup and have it running.
Now, I started backing up different folders with different scheduling to two separate repositories: I simply connected the UI to two repos.
I do this, because I want some important files at different locations (my local USB HD and a cloud storage) - but not all files should got to the cloud (to save repo volume there).
I seems that the server that kopia UI is starting uses quite a bit of memory after running the backups (about 1.2 GB), in fact it now has two servers running all the time.
So I wondered if I could simply run the backups via (systemd) timer.
But as I already defined all the policies I need in kopia, I wonder if I could just start a kopia server and tell it to stop again after all scheduled tasks have been executed (or run all pending backup tasks without starting a server)? Is there such an option already?
We discussed adding something like this, it would basically automatically register scheduled tasks with crontab or systemd timers on Unix/Linux or use Windows task scheduler.
If anybody’s interested in implementing this, I can help/guide through the process.
In the first place I thought about a simple option for kopia to do such a run so I could do the scheduling myself.
But of course, it would be nice for new users to have all setup automatically. I guess it’s a quite common use case.
I would like to invest some time here, can’t promise when and how much. But I’d like to give it a try.
I guess the first step would be to implement such an option with the CLI and API?
I never used Go before and only had a very brief look at the server sources, but it doesn’t look too complicated… So I’d be glad if you can provide some hints where and how to start.
I investigated crontab-based approach and it’s really only going to work (barely) on Linux, as it turns out macOS and Windows have some limitations that make this approach non-workable.
My current thinking is to install a scheduling service running as the current user, which will be just like kopia server start except without any UI or API.
Ideas for usage:
kopia scheduler install | start | stop | uninstall - will install/start/etc. OS-specific service (systemd on Linux, Win32 service on Windows, launchd on macOS) that runs at OS startup and keeps taking snapshots at predefined times until it’s stopped.
We can even integrate this at kopia repository connect | disconnect based on a policy setting or a flag, so:
$ kopia repository connect ... --install-scheduler
$ kopia repository disconnect # will remove any scheduler service
There are all kinds of security considerations here (keychain is generally protected by user password, and we still want to start the service without user having logged in, I think.
I’d actually prefer an interative bottom-up approach:
First provide the possibility to run the schedules snapshots. This could be used on any system already and we can provide help in the documentation how to schedule this on Windows and with cron or systemd timers.
Then, we can get feedback of others as well how they use it and what works best on different systems to do the scheduling and what are important and useful things.
The question WRT how to handle different repositories is important. Myself I also do backup to two different repositories (as I wrote above). And I noticed that using the UI, both servers start their hourly backup exactly at the same time and this sometime slows down the system, in spite of nice and ionice (I guess because of high memory usage, didn’t check yet). So here, I think it’s important to provide enough flexibility for different use cases.
And in future iterations maybe some “intelligence” as well. Something like: choose the compression depending on the currently available memory. Or allow to postpone snapshots while also free memory is low. … but in future.
So I would like to give something like kopia snapshop run-scheduled a try.
I haven’t looked at the server code yet. And would be glad to get some pointers for the things I would need - if you agree that this is a good strategy.
My thoughts so far:
I guess I can see how to get the list of registered snapshots by looking into what the UI is requesting and how the server handles this request.
Once I have the list of snapshots, I can iterate it and see which of those are due or overdue (as this is shown in the UI, I should be able to find out how to get this information) - and then run those (which should be simple, I assume I just need to invoke existing functionality).
I guess, we should do some filtering to the current user and host that is executing the process, to keep snapshots consistent. I assume that it won’t be a good idea to try to run a snapshot in a different context.
Next step: When this is run as root (on Linux), we could think about providing an CLI / API option to su to the different users on this system and start all snapshot tasks for all of them one after the other. Such as --run-for-all-users. Name is not ideal, it should say that the snapshots are actually taken as different users.
Another possible extension or way to achieve the same result: add some CLI / API option to filter for specific snapshots only (by user, machine, path).
What do you think?
Sounds like a task that could be possible for a kopia and Go newbie…?