Very slow commands on rclone Google Drive repository

Support
1

I have two Kopia servers running in separate docker containers on a Debian machine. They both backup the same data but one points to a repo on a machine on my LAN via SFTP and the other points to an rclone Google Drive repo.

The total backup size is around 500GB. I noticed as those 500GB were slowly uploaded to the rclone Google Drive repo that operations on that repo became very slow.

For example, just running kopia repository status on the two repos takes:

  • 0m0.793s for the local SFTP repo
  • 0m28.467s for the rclone Google Drive repo.

If I look at kopia maintenance info, I can see that as the the repository increased in size over the last few days (as the data was uploaded), the time for maintenance operations ballooned up to 40+ minutes for things like “full-delete-blobs” and “full-rewrite-contents”, whereas for the local repo it says 2-6 seconds. Just executing the kopia maintenance info command takes 20 seconds. All other commands that touch the repo are similarly slow (like updating policies, etc).

Is this expected on an rclone Google Drive remote or is my repo in some sort of super fragmented state? Is there anything I can do to speed it up?

1 Like
2

I’m on the same (slow) boat. Trying to run

kopia repository sync-to --parallel=32 filesystem --path=Z:\Kopia_001_replica

with a GDrive (Rclone) repository (4.5 TB) it take days just to complete the “Looking for BLOBs to synchronize…”. And when finally the sync (copying) starts, the speed is ~35KB/s. So, it would take ~5 years to transfer the data.

rclone copy --transfers 8 from the same repository sustains 9.5 MiB/s+ speeds. I would still take around 6 days to complete, but feasible.

I’m doing that on a Windows 11 Powershell prompt.