# When moving a repository: simple copy, \`repository sync-to\` or \`snapshot migrate\`? **URL:** \ **Category:** General Topics\ **Created:** [November 18, 2024, 1:20pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453 "2024-11-18T13:20:05Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![kopund](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kopund](https://kopia.discourse.group/u/kopund)\ **Post date:** [November 18, 2024, 1:20pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/1 "2024-11-18T13:20:05Z")
tl;dr: I want to move a 5TB repository from a WebDAV location to a local hard drive. Should I do a simple file copy, use `repository sync-to filesystem` or `snapshot migrate`? More detail: I had to move a repository before, and I think I might have done it incorrectly. So before I go through the process again, I would like to make sure I don’t mess up this time. 😂 I have a 5 TB repository that I originally copied from Google Drive to a local external hard drive using rclone. Now this hard drive is connected to a friend’s NAS running Nextcloud, and my kopia backs up to it via WebDAV. I have found many errors on this repo, so I’m planning to move to a local repository that is `repository sync`’ed to an offsite location, hoping this will be a more stable setup that also allows for verification (which WebDAV does not). Should I simply copy the files using e.g. rsync, should I use `repository sync-to filesystem`, or should I use `snapshot migrate`? I believe the first two options should yield the same result, while the third option would create a completely fresh repository? Any advantages or disadvantages to either approach or is there something I have not considered? ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [November 19, 2024, 7:13am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/2 "2024-11-19T07:13:33Z")
Repo inherent errors won’t magically vanish just because you copied the repo to another storage unless there’s something wrong with the connection between your Kopia client and the repo. That being said, I’d only use WebDAV as some kind of last resort, since WebDAV has inherent performance issues when it comes to the handling of lots of files /folders. I’d also always recommend to use Kopia’s sync-to feature, since its generally the fastest way to move and update a repo to another location. What errors are you experiencing? ---
**Author:** ![kopund](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kopund](https://kopia.discourse.group/u/kopund)\ **Post date:** [November 19, 2024, 10:13am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/3 "2024-11-19T10:13:31Z")
You’re of course right, errors won’t just vanish. But I’ve already gone through an odyssey of fixing these errors and the last one has just been fixed by Kopia 0.18, yay! (If you feel like going into it, [here](https://github.com/kopia/kopia/issues/3869) and [here](https://github.com/kopia/kopia/issues/3563#issuecomment-2143385151) are two of the posts about my issues.) I _think_ the errors stemmed from the Google Drive time of the repo, but I’m not sure. I also have no idea what might have caused them. I might have some missing files in my repo now if I’m not mistaken. But this is the best I have now, aside from throwing the backup history out the window. The main motivation to move the repo again comes from me wanting to be able to run verification regularly, which is unfeasible using WebDAV (you helped me figure this out [here](https://kopia.discourse.group/t/which-is-better-webdav-or-kopia-server-on-remote-nas/3248/11)). But since WebDAV is my only option (for now) for the offsite copy of my repo, I was going to follow the advice of having a local repo that is verified daily and then synced to the offsite location using WebDAV. I hope WebDAV will be good enough for that - so far it has actually worked really well for the daily backup run, just not for verification. Now that I’m getting into the details, the full story is actually that my current repo _still_ has all those 32179 errors. I had copied it over to an external hard drive to be able to experiment with that copy. The copy is now error free (see first paragraph above). But my actual repo is not. So now I see two paths forward: 1. bring the hard drive containing my current repo back in-house, fix all errors, and `repository sync-to` it to the offsite location via WebDAV, or 2. bring the hard drive containing my current repo back in-house, `snapshot migrate` it to a new repo on another local hard drive, and `repository sync-to` that new repo to the offsite location using WebDAV. Path 2 would, if I understand @jkowalski correctly [here](https://kopia.discourse.group/t/dealing-with-snapshot-corruption/767/2), eliminate all errors in the process - basically by dropping (and hence missing) some files. Which my error fixing probably also does, doesn’t it? Sorry, I tried to be succinct, but this long text is the best summary I came up with. 🫣 ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [November 20, 2024, 8:14am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/4 "2024-11-20T08:14:56Z")
Well… better too much than too few information, right? 😉 If option 2 would get your current repo up to speed and error free, I’d go for that. ---
**Author:** ![kopund](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kopund](https://kopia.discourse.group/u/kopund)\ **Post date:** [November 20, 2024, 9:41am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/5 "2024-11-20T09:41:14Z")
Thanks, I’ll try that and will report back here! Ah, one more question: if I `sync-to` via WebDAV, it’ll take two weeks or so. Can I still run backups during that time or will that interfere with this long-running `sync-to`? Ah, one more 😉: I assume (haven’t looked into it properly yet) that once everything is up and running, the daily `sync-to` should run automatically after every backup, right? I would probably not turn that on until the original `sync-to` is finished. But that made me think: what happens if there are e.g. connectivity issues and the daily `sync-to` doesn’t finish until the next daily `sync-to` begins? ---
**Author:** ![IcePic](https://avatars.discourse-cdn.com/v4/letter/i/74df32/32.png) [@IcePic](https://kopia.discourse.group/u/IcePic)\ **Post date:** [November 20, 2024, 10:34am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/6 "2024-11-20T10:34:55Z")
It should not be super hard to first see that no “sync-to” is running, and if not, then you run sync-to. ---
**Author:** ![kopund](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kopund](https://kopia.discourse.group/u/kopund)\ **Post date:** [November 20, 2024, 12:23pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/7 "2024-11-20T12:23:10Z")
By “see”, do you mean write some condition that `sync-to` only runs if there is no active `sync-to` process or check manually? Checking manually is not a sensible approach in an automated backup system. ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [November 20, 2024, 5:07pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/8 "2024-11-20T17:07:35Z")
First off… running a new snapshot while sync-to is also running shouldn’t matter. You can think of a sync-to like an rsync equivalent for Kopia. First it evaluates both sides and then crafts a plan on what to sync. Should a new snapshot be created, this will not be noticed by the running sync-to and the new data, will be taken care of the next run. The only thing I disable, while running a sync-to operation is the repo maintenance, since that can cause blobs to get deleted and should the sync-to job still expect these blobs to be there, you will run into an error. ---
**Author:** ![kopund](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kopund](https://kopia.discourse.group/u/kopund)\ **Post date:** [November 21, 2024, 11:48am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/10 "2024-11-21T11:48:55Z")
Huh… I accidentally deleted my post! So here comes again: Thank you @budy, understood regarding concurrent backup and sync-to. But what about two concurrent sync-to processes? My idea was to run sync-to as an After Snapshot Action. The backup runs every morning, so sync-to would also run every morning. If for whatever reason, sync-to is not finished by the next morning’s automatic sync-to beginning, do I run into trouble? Or is it not recommended to run sync-to as an After Snapshot Action? Oh and regarding maintenance: am I understanding correctly that I should write a script to turn off maintenance, then run sync-to, and then turn on maintenance again? ---
**Author:** ![IcePic](https://avatars.discourse-cdn.com/v4/letter/i/74df32/32.png) [@IcePic](https://kopia.discourse.group/u/IcePic)\ **Post date:** [November 21, 2024, 1:17pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/11 "2024-11-21T13:17:25Z")
Well, I mostly run kopia from crontab on unix systems, so in that case some version of “pgrep” or “ps -elf |grep kopia …” would tell me if there is another sync-to running or not. Some crontab implementations also have special ways to tell it only to run a line in the crontab if it isn’t already running. I’m sure windows has something similar one could use to notice programmatically if a program with “sync-to” in the name/commandline is running or not. ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [November 21, 2024, 5:05pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/12 "2024-11-21T17:05:43Z")
Again, you only need to worry about maintenance kicking in… well and probably having running multiple sync-to job over one another really would be “sub-optimal” in regards to performance, since sync-to would never delete anything from the local repo, I’d still assume this to be a non-issue. As long as your only having one client delivering snapshots to your repo, I’d say having the sync-to running as a After Snapshot Action would be ok, as long as you’ d also halt the maintenance for that repo. I’d suggest to simple perform the following steps in the After Snapshot Action: - halt maintenance - perform sync-to job - enable maintenace If you’re having multiple clients backing up to this repo that solution is probably not the best. Since I have 5 clients backing up to this repo, I have setup a cron job for the sync-to operation, which is likely to always run when no other client is using the repo. Admittedly, there have been instances, where the sync-to was not successful - e.g. when a client had to backup a lot of new data, but that would be resolved at the next run, so that is no big deal for me. ---
**Author:** ![kopund](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kopund](https://kopia.discourse.group/u/kopund)\ **Post date:** [November 22, 2024, 12:58pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/13 "2024-11-22T12:58:33Z")
Cool! I put this in a script, seems to work, if you see any mistakes I would appreciate feedback, otherwise maybe it will help some other noob in the future: ```auto #!/usr/bin/bash # Turn maintenance off kopia maintenance set --enable-quick=false --enable-full=false && \ # Start repository sync-to kopia repository sync-to webdav --url="https://URL" --webdav-username="username" --webdav-password="password" --delete && \ # Turn maintenance back on kopia maintenance set --enable-quick=true --enable-full=true ``` One more thing I’m wondering about is whether it would be OK to turn off automatic maintenance generally and just run it manually as a Before Snapshot Action. (I asked about it [elsewhere](https://kopia.discourse.group/t/details-of-maintenance-command/54/14), but that thread is quite old, not sure it will get a response.) That way, the repo hard drive, which is a simple external USB drive, would only have to jump on once a day, do quick maintenance, then full maintenance, then the daily backup run, and then go back into standby. It might survive longer that way than if it jumps on every hour for 10s. ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [November 22, 2024, 10:44pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/14 "2024-11-22T22:44:52Z")
I can see nothing wrong with your script. However, I’d rather have Kopia take care of the maintenance and not run it as a before action. Partially, since you can never know, how long it will take and that may delay your snapshot. Probably not a big deal, but I’d rather have my snapshots taken at the times I specified. ---
**Author:** ![dimejo](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/dimejo/32/287_2.png) [@dimejo](https://kopia.discourse.group/u/dimejo)\ **Post date:** [November 23, 2024, 10:59am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/15 "2024-11-23T10:59:48Z")
> [@IcePic](#): > > Well, I mostly run kopia from crontab on unix systems, so in that case some version of “pgrep” or “ps -elf |grep kopia …” would tell me if there is another sync-to running or not. Some crontab implementations also have special ways to tell it only to run a line in the crontab if it isn’t already running. Preventing concurrent script execution is the perfect use case for a tool like `flock`. ---
**Author:** ![IcePic](https://avatars.discourse-cdn.com/v4/letter/i/74df32/32.png) [@IcePic](https://kopia.discourse.group/u/IcePic)\ **Post date:** [November 25, 2024, 9:58am UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/16 "2024-11-25T09:58:31Z")
Yes, that is also an alternative. There are tons of ways, it is an old problem and many solutions were invented long ago. Might differ between OSes, but should be easy to find. ---
**Author:** ![kopund](https://avatars.discourse-cdn.com/v4/letter/k/e79b87/32.png) [@kopund](https://kopia.discourse.group/u/kopund)\ **Post date:** [November 26, 2024, 2:25pm UTC](https://kopia.discourse.group/t/when-moving-a-repository-simple-copy-repository-sync-to-or-snapshot-migrate/3453/17 "2024-11-26T14:25:09Z")
Thanks everybody! Will test it all out and report back in a bit.