Kopia sync-to rclone using webdav

I think the main difference is, as you mentioned in the other post, the initial check. Direct rclone command does not do that check. Also, I specifically used --size-only to speed up the checks, assuming the filenames already captures the hash of its content.

Another reason might be that sync-to rclone has --parallel=1 as default, while rclone does not (I usually see 4 files simultaneously being processed). This difference is amplified when user’s network bandwidth is capable of carrying the full workload (I have 1Gb fiber network).

So I just tried uploading my local repo to a new remote directory on Microsoft OneDrive. Since it is new, the initial check is quickly finished. I used --parallel=4 and I see great improve on the speed. However, at the first run, the upload quickly terminated with a weird error saying "kopia.exe: error: error copying blobs: error copying n639351c72e5260f3423a20ce0085a563: error writing blob 'n639351c72e5260f3423a20ce0085a563' to destination: BLOB not found, try --help". I retried exactly same command. It went further and then stopped again with same error on a different blob.

I also remember there used to be a different but similar error. It happens randomly to one file. After retry it passes the first file then err at another random file. Rinse and repeat.

So I think my concern of using webdav is actually not performance, but rather usability. I don’t know why Kopia sync-to clone constantly gives me weird errors (there’s also the hanging at end that mentioned in the OP). rclone itself never has problem like that. So it could be some bug in Kopia?

FYI:
the type of rclone backend?
Microsoft OneDrive

rclone version
rclone v1.54.0
os/arch: windows/amd64
go version: go1.15.7

OS (I presume Windows)
Windows 10

the speed of your network link?
1Gb/s up and down

what kinds of speeds you’re seeing using kopia and rclone
With default --parallel=1, 20Mbit/s. With --parallel=4, I’m seeing 51 Mbit/s and still climbing before the error

how big is your repository (can be measured using kopia blob stats and kopia content stats)
I’m testing with a small 3GB repo

1 Like