New user here, coming from restic and trying to experiment with different solutions before settling for one.
I have several different parts of the filesystem (Ubuntu, if it matters and the parts would be /opt for docker setups, /home for obvious reasons and /etc for configurations that I need backed up plus different folders from /mnt/smb…) that I backup to different repositories (larger files only to local NAS/backup and personal files also to remote backup).
If I understood this correctly, when I first create a main repo, this gets saved (including a hashed version of the password) into ~/.config/kopia. Whenever I now use kopia snapshot /path, the main repo will be used.
What about other repositories, do I need to specify this via command line with kopia snapshot --config-file=~/.config/kopia/wasabi.config [...]? How can I sepeficy file/folder exlcusions with the CLI?
What about the maintenance tasks, will these be carried out only on the main repo or on all previously connected repos as well? Can I simply schedule kopia maintenance run it with a cron/systemd job.
As you can see, I am a bit confused about the mix of GUI and CLI usage. As I mainly back up server files, a CLI with a simple config file would be best.
Yes, using different config files you can have multiple repositories.
Just keep in mind, if you would do backup to different repositories then all of them will be un-synchronized since file changes might occur between snapshots and of cause it would take the same amount of time to backup to different repositories. To avoid it, you can do backup to local (external drive/ local NAS) repository and late use sync-to command to synchronize freshly taken backup to other repository. This will give you synchronized (all repos will be identical) backup and this operation works much faster due to sync-to works kind of like rsync (syncing only unmatched parts on both sides). Just one notice regarding this workflow, - sync-to do not verify nor doing maintenance on target repositories, so in case you want be sure everything is Ok, you can run periodically via cron maintenance on those repos and may be verification if you not sure about quality of remote storage.
Maintenance executed automatically after each backup session, so basically not needed, but you can enforce to run it if you want, but as I said previously, it make sense for repos that was synched.
Yes.
GUI is just a wrapper on top of kopia, so no any advantage except to simplify backup for GUI people. You can get the “GUI” via browser if you would access kopia when it runs in server mode.
Thanks for your explanations, that was very handy!
Regarding the use of multiple repositories, I don’t want them to be synchronised as they receive different folders, so no problem there.
I managed to use the GUI now and while it was simple to set up new snapshots, set policies (excludes, schedule, compression, etc.) I am now still a bit at a loss with regards to where all this is stored (in ~/.config/kopia, only the …update.json from the previously used repo is present). I can change repositories in the UI, but only if I re-enter all data anew, as if creating a new repo. Is this the expected behaviour or is there some way to quickly change repos in the GUI? Is this dependent on the GUI user? Or should I only use the GUI to create the config-file and then use the CLI?
If I create the snapshot in the GUI, who/what is responsible to actually perform the snapshot, is this somehow added as a cronjob/systemd timer? Or is it necessary to run kopia server the whole time?
Additionally I had problems reading several files from /etc but I gave the kopia binary permissions to do this via sudo setcap cap_dac_read_search=+ep /usr/bin/kopia - not sure if this is the correct way, that’s what I used with restic.
kopiaUI is limited wrapper and oriented for simple workflows. If you want full power then you have to use kopia.
kopiaUI internally runs kopia server that use current user and host name, but you actually can override it.
No, as I said, kopiaUI is just wrapper and use internally kopia CLI. Basically it forms command line and passing it to kopia CLI. run ps aux and you will see it.
You can create as many configs as you need by using only kopia CLI.
kopia. CLI version.
No. When you run kopiaUI, it start kopia as a server and as far as it running in background, it can perform scheduled tasks.
Exactly. If you would choose workflow where all you need is taking snapshots, you can write your own scripts that you would run from cron job/systemd timers just for taking snapshots at particular time you need where kopia will run at scheduled time and terminate when it done.
Make you life easier, all backup solutions running with highest privileges exactly for this reason, - to be able to backup everything, so run it from /etc/cron.d/kopia-backup as a root user. setcap utility would work too, but keep in mind that you need to run it every-time you upgrading kopia.
We all been there
kopia is pretty powerful and well designed tool. You can use it for a simplest backups by utilizing just create/snapshot commands and it would be enough, but it also can run in server mode where you can push backups from multiple computers, use fine grained permissions such as for example activate APPEND ONLY mode to prevent malicious code delete or modify backups.
Try to run kopia --help-long >kopia.help.txt, you will find much more about this program. It isn’t simple, but when you get concept behind it, you will have pretty powerful tool.
By right-clicking the tray icon, you can add more repos to connect to, and all the connected repositories will then be available through the tray icon menu.
Sorry that I was not more clear but I am using the web UI and I don’t see a way to switch repos there. As Jarek said elsewhere, it is probably only possible by starting several instances of the server, is kind of cumbersome. I will try to refrain from using the web UI, which is a little sad as it was one of the main draws of kopia…
In general, currently Kopia is not designed to facilitate the same policy/snapshot to multiple repos. It is designed with a one-to-one relationship in mind, in the sense that policies/snapshots are associated with one repo only.
The way to use Kopia with multiple repos is to start multiple different instances of Kopia – this holds true regardless of whether you use Kopia CLI, Kopia web-based GUI, or Kopia desktop-based GUI. It is just that the desktop-based GUI makes the process slightly easier due to the right-click menu that gives you the option right there.
Oh, I wasn’t clear again, should have explained my use-case in the first place…
I do not want to use the same policy/snapshot for different repos, I just use e.g. the local NAS as a repo for large files that could eventually be retrieved from somewhere and are not critical, but use a cloud backup with redundancy as a repo for irreplaceable data (family pictures for example). So, no two repos for the same data, it’s for different data sets.
Thanks, slowly getting there.
The main obstacle I still have (coming from restic/resticprofile) is that there is no clear configuration file that is used for backing up a snapshot/set of folders to a defined repo. There is a config file that is generated upon connecting to a repo (the json snippet that is also shown in the web UI), but it is removed after disconnecting, this is confusing for me.
I think that kopia could benefit greatly from a single config file per snapshot but then again, probably it is even more flexible/productive than restic’s way of doing, I just can’t wrap my head around it then…
You can do it when you stop doing Windows things on Linux
IMHO you over complicating your workflow by trying “intuitive” way with GUI
Connect to each repositories with different path to config file
Use CLI kopia to save token (read as config+password)
kopia repository status -t -s | awk '{print $7}'
save token and later use it as
kopia repository connect from-config --token ${reconnect_token}
#do snapshots
# disconnect
# connect to another repo with different token
# snapshot
...
Another way around, is to use environment variable KOPIA_CONFIG_PATH in each script that do snapshots to a different repo, so each of script will use particular repository
This way you don’t need kopia server or GUI wrapper, create connection to repositories with different config paths then simply use it in snapshots for each repository.
Yes, that is what I am trying now, started by writing a small bash script to first connect and then backup and do maintenance. However, I find it difficult to even find out how to exclude files from a snapshot. I know it can be done via the UI but I want/need it to be done by command line. According to the documentation, this is done by setting the appropriate policy per snapshot but where can I find details on how it is done?
I can e.g. kopia policy edit --global, which throws me to vim with a temp copy of the global policy. Where is this policy stored and how can I edit it with the editor of my choice (micro)? How can I save a policy to a config file and reference it in a snapshot command or where can I find a reference on what to set in my script via kopia policy set? Setting up a proper backup strategy takes work and I would like to know that I can recreate this workflow someplace else and not rely on a policy that is stored somewhere…
I am terribly sorry if this comes across as whiny, but I truly would like to understand and I have the idea that a backup regime with kopia could be potentially great but currently I am not (yet) getting it.
I believe I pointed you out to github link that describes using .kopiaignore (basically it has the same rules as .gitignore), so placing it in snapshot directory and .kopiaignore will exclude files you set there or you can do the same via policy (multiple --add-ignore in kopia policy set …)
kopia policy set --global --config-file /path/to/config/file/for/particular_repository --Policy1 --Policy2 ...
Where policy are options from here: policy set | Kopia
And -global means “global” for all snapshot targets on that particular repository described with particular config. (You you can omit --global and set policy per snapshot in that particular repo)
Do not use low level editing of json file unless you deeply understanding where it come from.
Use instead policy set/delete with appropriate option from link above that you want to setup on particular repository (represented via its config).
Answering your question, - policy kept in repository, that’s why all you need to know about backup, it is location and password, so you can connect to it from any place. If kopia would keep policy locally in a config, then this advantage will be gone and in case source machine that you backing up would die, then it lost backup policy which I think you would agree - is un-logical in case of backup.
I hope all other questions now looks more clear , but anyway, if you still need help, then continue to ask.
Thanks again for your explanations, especially the one about where the policies are kept helped a lot to understand kopia’s design principles!
One thing I was looking for is a list of policies to set. For example I wanted to set the ignorefile to .kopiaignore (thanks for the Github link, must have missed it before) but all I could find was the flag --add-dot-ignore - is this the correct one? The .kopiaignore files need to be placed in the respective snapshot targets, right (e.g. if I were to snaphot /etc, the dotfile needs to be put at /etc/.kopiaignore)?
