Rclone restores very slowly

Support
1

Hi,

I’m trying to use kopia with a rclone onedrive remote. The backup speed is pretty fast, but the restore is taking ages. For a 200MB snapshot, it’s still ongoing after 25 minutes.

Using rclone sync, I was able to copy the kopia repo to my local filesystem in less than 30 seconds.

My repo is mainly small files and some git repos.

Is there any way to make this faster?

1 Like
2

Kopia does not (yet) utilize any parallelism during restore (unlike upload which is heavily parallelized) . This means that with a high-latency store (like OneDrive) you may see long restore times, in particular when you have tons of small files. Parallel restore is coming in future versions, for now if you need to restore a lot of small files quickly. syncing the entire repository locally and restoring from there is the recommended approach.

I’ll prioritize pipelined restores for 0.8.

1 Like
3

Thanks for the info. I guess it’s not a big deal to me since I’ll have at most 10-20GB backed up. But it may be a problem for people with terabytes backed up.

Here’s the steps followed for a fast restore in case anyone else comes across this:

rclone sync onedrive:/backups/kopia kopiarepo
kopia repo connect filesystem --path kopiarepo
kopia restore <snapshot id> kopiarestore

On a side note, “rclone sync” was 7x faster for me than “kopia repo sync-to”.

1 Like
4

Thanks for the info. I forgot to mention that sync-to takes --parallel parameter which defaults to 1, which won’t be the fastest. Can you try kopia repo sync-to --parallel=32 filesystem --path=/some/path ?

Perhaps we should change the default…

5

With parallel set to 32 it’s finishing in 64 seconds, compared to 150 before. That’s good, but rclone sync is still faster (20 seconds).

I think it’s because it spends abit of time in the “Looking for BLOBs to synchronize” bit, whereas rclone gets straight to copying.

6

Yes, I think the default should be changed.

I am trying to restore from an rclone remote and for a 14 GB snapshot to restore it was going to take 155 hours. That’s really excessive.

I changed it to parallel 32 and started over, it finished in under 10 minutes.