tl;dr:
I want to move a 5TB repository from a WebDAV location to a local hard drive. Should I do a simple file copy, use repository sync-to filesystem or snapshot migrate?
More detail:
I had to move a repository before, and I think I might have done it incorrectly. So before I go through the process again, I would like to make sure I don’t mess up this time.
I have a 5 TB repository that I originally copied from Google Drive to a local external hard drive using rclone. Now this hard drive is connected to a friend’s NAS running Nextcloud, and my kopia backs up to it via WebDAV. I have found many errors on this repo, so I’m planning to move to a local repository that is repository sync’ed to an offsite location, hoping this will be a more stable setup that also allows for verification (which WebDAV does not). Should I simply copy the files using e.g. rsync, should I use repository sync-to filesystem, or should I use snapshot migrate?
I believe the first two options should yield the same result, while the third option would create a completely fresh repository? Any advantages or disadvantages to either approach or is there something I have not considered?
Repo inherent errors won’t magically vanish just because you copied the repo to another storage unless there’s something wrong with the connection between your Kopia client and the repo. That being said, I’d only use WebDAV as some kind of last resort, since WebDAV has inherent performance issues when it comes to the handling of lots of files /folders.
I’d also always recommend to use Kopia’s sync-to feature, since its generally the fastest way to move and update a repo to another location. What errors are you experiencing?
You’re of course right, errors won’t just vanish. But I’ve already gone through an odyssey of fixing these errors and the last one has just been fixed by Kopia 0.18, yay! (If you feel like going into it, here and here are two of the posts about my issues.)
I think the errors stemmed from the Google Drive time of the repo, but I’m not sure. I also have no idea what might have caused them. I might have some missing files in my repo now if I’m not mistaken. But this is the best I have now, aside from throwing the backup history out the window.
The main motivation to move the repo again comes from me wanting to be able to run verification regularly, which is unfeasible using WebDAV (you helped me figure this out here). But since WebDAV is my only option (for now) for the offsite copy of my repo, I was going to follow the advice of having a local repo that is verified daily and then synced to the offsite location using WebDAV. I hope WebDAV will be good enough for that - so far it has actually worked really well for the daily backup run, just not for verification.
Now that I’m getting into the details, the full story is actually that my current repo still has all those 32179 errors. I had copied it over to an external hard drive to be able to experiment with that copy. The copy is now error free (see first paragraph above). But my actual repo is not.
So now I see two paths forward:
bring the hard drive containing my current repo back in-house, fix all errors, and repository sync-to it to the offsite location via WebDAV, or
bring the hard drive containing my current repo back in-house, snapshot migrate it to a new repo on another local hard drive, and repository sync-to that new repo to the offsite location using WebDAV.
Path 2 would, if I understand @jkowalski correctly here, eliminate all errors in the process - basically by dropping (and hence missing) some files. Which my error fixing probably also does, doesn’t it?
Sorry, I tried to be succinct, but this long text is the best summary I came up with.
Ah, one more question: if I sync-to via WebDAV, it’ll take two weeks or so. Can I still run backups during that time or will that interfere with this long-running sync-to?
Ah, one more : I assume (haven’t looked into it properly yet) that once everything is up and running, the daily sync-to should run automatically after every backup, right? I would probably not turn that on until the original sync-to is finished. But that made me think: what happens if there are e.g. connectivity issues and the daily sync-to doesn’t finish until the next daily sync-to begins?
By “see”, do you mean write some condition that sync-to only runs if there is no active sync-to process or check manually? Checking manually is not a sensible approach in an automated backup system.
First off… running a new snapshot while sync-to is also running shouldn’t matter. You can think of a sync-to like an rsync equivalent for Kopia. First it evaluates both sides and then crafts a plan on what to sync. Should a new snapshot be created, this will not be noticed by the running sync-to and the new data, will be taken care of the next run.
The only thing I disable, while running a sync-to operation is the repo maintenance, since that can cause blobs to get deleted and should the sync-to job still expect these blobs to be there, you will run into an error.
Huh… I accidentally deleted my post! So here comes again:
Thank you @budy, understood regarding concurrent backup and sync-to. But what about two concurrent sync-to processes?
My idea was to run sync-to as an After Snapshot Action. The backup runs every morning, so sync-to would also run every morning. If for whatever reason, sync-to is not finished by the next morning’s automatic sync-to beginning, do I run into trouble?
Or is it not recommended to run sync-to as an After Snapshot Action?
Oh and regarding maintenance: am I understanding correctly that I should write a script to turn off maintenance, then run sync-to, and then turn on maintenance again?
Well, I mostly run kopia from crontab on unix systems, so in that case some version of “pgrep” or “ps -elf |grep kopia …” would tell me if there is another sync-to running or not. Some crontab implementations also have special ways to tell it only to run a line in the crontab if it isn’t already running. I’m sure windows has something similar one could use to notice programmatically if a program with “sync-to” in the name/commandline is running or not.
Again, you only need to worry about maintenance kicking in… well and probably having running multiple sync-to job over one another really would be “sub-optimal” in regards to performance, since sync-to would never delete anything from the local repo, I’d still assume this to be a non-issue.
As long as your only having one client delivering snapshots to your repo, I’d say having the sync-to running as a After Snapshot Action would be ok, as long as you’ d also halt the maintenance for that repo.
I’d suggest to simple perform the following steps in the After Snapshot Action:
halt maintenance
perform sync-to job
enable maintenace
If you’re having multiple clients backing up to this repo that solution is probably not the best. Since I have 5 clients backing up to this repo, I have setup a cron job for the sync-to operation, which is likely to always run when no other client is using the repo. Admittedly, there have been instances, where the sync-to was not successful - e.g. when a client had to backup a lot of new data, but that would be resolved at the next run, so that is no big deal for me.
Cool! I put this in a script, seems to work, if you see any mistakes I would appreciate feedback, otherwise maybe it will help some other noob in the future:
#!/usr/bin/bash
# Turn maintenance off
kopia maintenance set --enable-quick=false --enable-full=false && \
# Start repository sync-to
kopia repository sync-to webdav --url="https://URL" --webdav-username="username" --webdav-password="password" --delete && \
# Turn maintenance back on
kopia maintenance set --enable-quick=true --enable-full=true
One more thing I’m wondering about is whether it would be OK to turn off automatic maintenance generally and just run it manually as a Before Snapshot Action. (I asked about it elsewhere, but that thread is quite old, not sure it will get a response.) That way, the repo hard drive, which is a simple external USB drive, would only have to jump on once a day, do quick maintenance, then full maintenance, then the daily backup run, and then go back into standby. It might survive longer that way than if it jumps on every hour for 10s.
I can see nothing wrong with your script. However, I’d rather have Kopia take care of the maintenance and not run it as a before action. Partially, since you can never know, how long it will take and that may delay your snapshot. Probably not a big deal, but I’d rather have my snapshots taken at the times I specified.
Yes, that is also an alternative. There are tons of ways, it is an old problem and many solutions were invented long ago. Might differ between OSes, but should be easy to find.
If option 2 would get your current repo up to speed and error free, I’d go for that.