Speed up sync-to compare repo

General
1

Hello,

Kopia repository sync-to is a key keep offsite backups in sync while this task takes no time to compare source and destination for the first sync.

The sync-to compare operation takes lots of time for late resync depending on the size of the source repository. In case of small changes in the source repository few megabytes the comparison of the gigantic repository takes far longer time than downloading the changes to the destination.

Can something be done to improve comparison speed?

Regards,
Sd

2

Not sure what you mean. sync-to works fairly quickly for me (<1m even with changes to the original repo, although the changes to the original repo thus far for me have been <100mb). Have you checked your logs to see what is delaying it?

It would also help if you define what you mean by “far longer time”.

3

The line…

Found x BLOBS (x.xGB) in the source repository, xxxx (xxGB) to copy

This operation takes longer.
Using rclone remote.

4

I don’t know what you mean by “takes longer”. There are many variables that impact how long it takes to copy blobs to another repo. There isn’t a a known bug in sync-to that is slowing things down, if that is what you mean. Have you tried using the --parallel to increase parallelism? It is 1 by default. Increasing it should dramatically decrease sync-to time, just be careful not to overload your target as some may limit the amount of concurrent connections.

5

In my last test for sync-to

Sync-to for a repository size of 300 GB took 31mins with no changes in the source repository on a 100mbps link.
I really don’t know if that is normal…

With–parrallel=64

6

Can you share the log for this sync-to procedure? What you are describing sounds unusual.

7

Sorry for the big delay
here is the log

2022-07-02T11:26:20.275606Z DEBUG cache finished sweeping {"cache": "metadata", "duration": "504.5µs", "totalRetainedSize": 0, "tooRecentBytes": 0, "tooRecentCount": 0, "maxSizeBytes": 5242880000, "inUsePercent": 0}
2022-07-02T11:26:20.276309Z DEBUG cache finished sweeping {"cache": "contents", "duration": "504.5µs", "totalRetainedSize": 0, "tooRecentBytes": 0, "tooRecentCount": 0, "maxSizeBytes": 5242880000, "inUsePercent": 0}
2022-07-02T11:26:20.276979Z DEBUG cache finished sweeping {"cache": "index blob cache", "duration": "1.877ms", "totalRetainedSize": 0, "tooRecentBytes": 11844031, "tooRecentCount": 41, "maxSizeBytes": 5242880000, "inUsePercent": 0}
2022-07-02T11:27:20.289152Z DEBUG cache finished sweeping {"cache": "metadata", "duration": "573.5µs", "totalRetainedSize": 0, "tooRecentBytes": 0, "tooRecentCount": 0, "maxSizeBytes": 5242880000, "inUsePercent": 0}



2022-07-02T12:37:21.129329Z DEBUG cache finished sweeping {"cache": "metadata", "duration": "546µs", "totalRetainedSize": 0, "tooRecentBytes": 0, "tooRecentCount": 0, "maxSizeBytes": 5242880000, "inUsePercent": 0}
2022-07-02T12:37:21.129329Z DEBUG cache finished sweeping {"cache": "contents", "duration": "1.1966ms", "totalRetainedSize": 0, "tooRecentBytes": 0, "tooRecentCount": 0, "maxSizeBytes": 5242880000, "inUsePercent": 0}
2022-07-02T12:37:21.368705Z DEBUG cache finished sweeping {"cache": "index blob cache", "duration": "2.1434ms", "totalRetainedSize": 0, "tooRecentBytes": 11844031, "tooRecentCount": 41, "maxSizeBytes": 5242880000, "inUsePercent": 0}

I find a lot of time going in sweeping starts at 11:26;20 goes upto 12:37:21. Then getblob beings ends around 12:56:11
I can’t upload the full log file here…

Regards,
Savio D’souza

8

The slowness of the process may be related to your hardware. Are you running HDDs or SSDs?

9

Source is Rclone remote onedrive
Destination local HDD

swapping is happening on rclone remote…

10

My guess is that this is why. HDDs are notoriously slow. Also, OneDrive via Rclone can also be slow because OneDrive has API limitations that it puts on non-business accounts (and Rclone itself also adds some delay to the whole process as compared to use a native Kopia repo).

Also, you said you used --parallel=64. That may be too much for your HDD to handle. Perhaps something lower, like 5 or 10 or leave it at the default setting.