First of all thanks for your time on responses and thanks for Kopia. I’m a new user of copia, but a user of restic and gocryptfs (similar approach). I saw some diff that I need to understand in advance (I didn’t found the answer on the forum):
Main difference between others and Kopia is “polices”. Normally, I need to run the backup command in order to launch the backup program. I saw that there is a Kopia server that can run this command for me following policy config. I don’t see how to run this server on OSx or Linux. I should do it with Crontab. Is this the correct approach? Then, why I need policies?
I understand that Kopia is designed for allow an encrypted backup on a public cloud. I mean, Kopia doesn’t leave hidden config files on the backend repository that can be used for breaking security of the repository and compromise the content?
Is there any limitation in terms of file size or repository size?. I’m looking for use Kopia with 2Tb of data, some files bigger than 3 Gb,
You have slightly misunderstood the server part of Kopia. The server part is only to allow remote Kopia clients to reach a repository, which they don’t host theirselves. It doesn’t perform any backups on its own - this is the task of the Kopia client. On macOS, the KopiaUI is able to perform automated snapshots, but only if it is running.
You may look into creating your own launch item for the Kopia CLI client, to perform fully-automated snapshots without KopiaUI being run.
No - there are no files stored in the repo, which would allow for anyone to reverse-engineer the encryption.
I don’t think, there’s such limits. I am using Kopia to backup a rather large Samba server with approx. 97T of content and surely, there are a couple of very big files on there as well.
About #1, if I understood correctly, I just need to define the number of snapshots I want to keep, and I manually run on client scheduled snapshots via crontab or similar… right?
#2 > Great… thanks a lot.
#3 > Perfect… thanks.
I need now to find out the way of scheduled backups and run them silent and in the background. I tried some crontab config, but I realise that it didn’t run because the osx was sleept …
Do you have a sample of scheduled snapshots on OSX?
Well, so far I have only used the KopiaUI app on macOS. f you wanted a launch agent, then there would be some fiddling to do. First thing I noted was that zsh, which is the default shell under Catalina, doesn’t take the --password parameter well and bails out - I have not worked around that, when I tried kopia purely on the CLI, but I can maybe come up with something - I’ll let you know.
And yes, the retention controls which snapshots are kept and it it independant of when or how you run Kopia to perform a snapshot.
This is actually the first thing I notice. I just need to put the password first time I created the repositor (in my case SFTP). I used SSH authentication via public key for SFTP… but I don’t put the pass on the kopia command (I assume that it’s stored anywhere on the Kopia Client OSx).
Normally (other programs) I need to pass the password for each snapshot… am I doing something wrong?
A part, I saw that Policies only applies to “server”. If I’m running directly client via OSx CLI… policies doesn’t apply?.
I don’t use SFTP, so I can’t provide any meaningful input, but as of Oct. last year, SFTP has been added to Kopia’s codebase, so the help should be of assistance.
Couple clarifications - kopia server (today) DOES run scheduled snapshots for the user that launched it.
It does three things:
serve repository API for remote users (which can use it to access their own data - contents and manifest). The set of users is provided via htpasswd.
serve UI to local user
runs scheduled snapshots for local user according to scheduling policies
KopiaUI leverages 2&3 and is simply a small electron shell around kopia, which does all the hard work.
I’m realizing that merging 1 with 2&3 was probably not the best idea (or at least it’s confusing). In next version we’ll have separate protocol for 1 (GRPC) vs the others (HTTP/REST) so it will probably make sense to split the server functionality in two and have two separate commands or flags for different purposes.
When it comes to scheduled backups both routes are supported:
crontab and launching kopia snapshot create --all periodically
starting kopia server and letting it do its thing whether it has remote repository users or not.
Policies apply when taking snapshots regardless of client vs server. You can specify things like compression, error handling, ignore rules, etc.
The only policy that does not make sense on the client is scheduling policy (because if the client is not running it can’t do anything about it). That’s why when kopia UI launches the server to serve the UI, it also acts as the long-running client, which takes snapshots periodically.
Repository connection parameters are stored in a local file (something line /Users/X/Library/Application Support/kopia/repository.config on a Mac, .config/kopia/repository.config on Linux and so on).
This specifies connection to blob storage.
On top of that there is a password to decrypt contents in blob storage. There are 3 choices:
it can be stored it in local keychain/keystore/Windows Credential Manager depending on the OS (this is the default)
it can be stored in a local file next to the config file
it can be not persisted at all - you must then pass it to kopia via KOPIA_PASSWORD environment variable on each run
Okay… so… basically, there is no way to have Kopia on the Mac run without the KopiaUI? See this thread about Kopia binary just crahing when trying to load an existing config file using the --file parameter:
Btw, I tried that on my Linux servers as well, which don’t even have a GUI, and that caused the same error… it’s rather confusing as I thought that Kopia and Kopia server, would somehow be the same binary.
Correct me if I’m wrong. I believe the server also runs scheduled maintenance automatically, if an interval is set.
Not sure about other OS, but on Windows since kopia.exe is a console app, it always starts a console window in foreground. It is really not fit for a background service. Of course I could kill the conhost.exe process to hide the console window, but I doubt any normal user would be able to do that by themselves.
Ideally, Kopia can provide a “service” executable that runs the server in background, log output to files by default.
EDIT: This post has some info about launching console app without conhost. For example, I just verified the DETACHED_PROCESS approach works and I haven’t noticed anything break. Worth investigating.
It should be possible to add native support for windows service quite easily. Support seems available in Golang standard library. I’ll be happy to review PRs to do it.
I have an issue on creating snapshots on OSx via Crontab, when the system is sleeping or is not active. I saw that snapshot has not been created. I don’t know how to force it happens on background even if the OSx is not “active”.
I already verified that this OSx is not entering on hibernation or sleep (it’s on a VM), but seems it’s not enough.
First thing I noted was that zsh, which is the default shell under Catalina, doesn’t take the --password parameter well and bails out - I have not worked around that, when I tried kopia purely on the CLI, but I can maybe come up with something - I’ll let you know.