Solution to slow rclone restore

General Topics
1

Related to Rclone restores very slowly and SOLVED 1) Backing up the cachingDirectory beneficial? 2) cacheDirectory inner working?

The Issue

Rclone downloads in 20 seconds, while kopia doesn’t seem like it will ever finish.
I took a look at --log-level=debug and it seems to be taking its time on copying .git folders which have many and very small files.

To make things worse, it seems to be using 3MiB/s to 14MiB/s of network in spikes to a microsoft ip, which is way more than the reported 600KB/s. I believe kopia is doing something very wrong for this to happen, like redownloading the repo for each file or something.

# rclone copy remote:Folder . --onedrive-chunk-size=25Mi --config=/run/secrets/rclone --progress --transfers=64
Transferred:   	 412.315 MiB / 412.315 MiB, 100%, 23.868 MiB/s, ETA 0s
Checks:                 1 / 1, 100%
Transferred:           40 / 40, 100%
Elapsed time:        18.6s

Kopia restore will never finish

The Solution

Use Rclone with VFS cache

kopia repo connect rclone .. \
	--rclone-args="--onedrive-chunk-size=25Mi" \
	--rclone-args="--vfs-cache-mode=full" \
	--rclone-args="--vfs-cache-max-size=1G" \
	--rclone-args="--cache-dir=/tmp/rclonecache"

This made my restore from probably 3+ hours to Executed in 141.63 secs!

Note

I tried --cache-directory /tmp/kopiacache --content-cache-size-mb 900 but kopia only used 4.8MB of that cache.

1 Like