I have a NAS running TrueNAS Scale which stores some important pictures and videos. I have split the files into a “small” and large part: The small part contains all pictures and compressed versions of the videos and the large one contains the original videos and additional non-important videos. I have done the split because my upload speed is not that great and should not hinder me to frequently backup the small part. There’s also an additional folder for documents and stuff but that should not be part of the discussion because those are not that important.
My strategy would be as follows:
Backup both parts to a portable USB SSD (“main backup”).
Copy both backups with kopia repository sync-to from the SSD to an HDD.
Copy both backups with kopia repository sync-to from the SSD to a B2 bucket.
I understand that sync-to could also be replaced by manually deleting all files on the target and re-copying everything but sync-to can incrementally upload files which is nice.
My main question is where do I run kopia to do the backups and lengthy uploads to B2? I have tried the following options:
Using the unofficial Helm-Chart / Container by TrueCharts on TrueNAS. Two instances/repositories, one for each part. I haven’t found a supported way of mounting the SSD into the container automatically. I have only done a backup directly to B2, skipping the sync-to. I have used the Web UI.
Using a separate machine (Raspberry Pi 1) in the same network as the NAS. The SSD is attached to the Pi. I also used the Web UI. Doing sync-to is somewhat manual via the terminal.
I am not happy with the solution.
I would like to trigger backups manually since I have to plug-in the SSD and HDD.
I would like to have a web based UI to start the backup and to start a sync-to.
I would like to make this setup as reproducible as possible.
Regarding 2: I could build a small web app to trigger some commands that are not supported by the OOTB kopia Web UI.
Regarding 3: I have started to build some scripts to setup a Raspberry OS instance from scratch to setup kopia. It asks the user for the token which was “generated” by kopia repository status reconnect-token-with-password to keep the user input minimal.
Do you have any advice? I certainly can build something but I am not sure if there might be an easier more proven way.
I know I’m answering a question you didn’t ask, but I would NOT sync your SSD to an external repository. If the SSD becomes corrupt, you are going to sync that to your second and third repo.
If you want to have 3 external copies of the backups, I would run three separate kopia containers (each with a different repository source).
I don’t know about TrueNas, but the general Docker advice would be:
docker compose run --rm
or
docker run --rm -d -v <volume, eg SSD> -v <volume, eg source files> kopia/kopia:latest snapshot create
As mentioned above, I would run manual snapshots. The command would be
docker run --rm -d v <volume, eg SSD> -v <volume, eg source files> kopia/kopia:latest server start --address=0.0.0.0:51515
The web UI will be available at the IP address of the box on port 51515
Docker compose. You can set it once and run it simply.
So just three repos that use the same snapshot source?
Even if I regularly run Kopias validation task (which I guess should detect corrupt repos) I might be better off with three separate repos because I don’t need to use the CLI to trigger a sync-to. I could manage everything in the web UI.
Currently I am thinking about installing Portainer as a “App” to my NAS. In the end Portainer itself is run as a container in TrueNAS but managed by TrueNAS. In Portainer I would have the three instances of Kopia as three containers. I guess I would need to build a Docker image myself, based on the Kopia image. The image needs to be able to mount the NAS folder (maybe via SFTP, NFS or SSH). To mount those, I would need to install some additional tools to the image.
The USB storage device might be made available via my router as a locally available FTP server. I never quite could get it to work to mount a USB device to a container on TrueNAS, that’s why I guess I need to resort to that solution. My router doesn’t support SFTP and in turn Kopia doesn’t support FTP. So maybe I need to mount the FTP folder into the container using something like curlftpfs.
Well, sounds like a complicated setup but what I most care about is that it is reproducible. Also the secrets to B2 and the repo secret can be stored in Portainer.
I don’t know when I will start implementing that idea. If I have something to share, I will post it here.