Troubleshoot slow uploads on decent connection

Support
1

Hi,

I’m trying to implement off-site backups with kopia, via tailscale, to a remote server. Connections are 500/100 and 1000/100mbps and iperf3 baseline tests show uploads of about 80mbps.
I’ve ruled out issues with tailscale by using direct sftp connection to the server.

I’m using macOS or Docker, with external ssh agent

I am experiencing slow transfer of blobs (logs below). From my understanding this is an upload of a blob of 28.15MB in 1m34s, which means a transfer speed of 2.34mbps, which is way below the 80mbps that I would be expecting.

DEBUG [STORAGE] PutBlob	{"blobID":"p252f06efeff37e13fa2f2723036cbee0-s5ceea3b21dbc74ac129","length":28156285,"error":null,"duration":"1m34.910299375s"}

My question is: the PutBlob only means actual transfer and this time does not include crypt, dedup etc?

On another note, to improve the overall speed of a snapshot, I have increased the --parallelism=100. However, it is not clear if this will set both --max-parallel-snapshots and --max-parallel-file-reads. Also, logs will show lines like DEBUG [STORAGE] concurrency level reached {"maxConcurrency":26}, which does not seem to be correct.

Any tips for troubleshooting the issue?
Thanks

2

I haven’t tinkered much with the parallelism settings, but this issue will not be on the source side. Depending on the type of remote storage, it may be advised to adjust the sharding, since kopia will choose different type of sharding for different type if storages.

Also, always take latency into account, when dealing with remote storages. The network bandwidth alone only determines the absolute maximum, it does not guarantee it.