I have recently setup Kopia with a Kopia repository server in TX that is connected to a backblaze S3 bucket.
While testing with other local client servers in TX I am able to achieve over 650mbps of upload through the machine and the hashing completes quickly.
I have then started to try and back up other servers from other locations, such as Virginia. When uploading these snapshots, the machine seems to top out at around 14mbps and does not go any higher.
I ran an IPERF to verify, and the speed between the client server and Kopia repo server is still ~800 Mbps.
Does anyone have any ideas what might cause such a large slowdown when all client servers are running pretty powerful hardware (Ryzen 7900s) and they are both under similar load?
My only hunch is that this could be caused by the slightly higher latency from Virginia to Texas. If this is caused by the latency, do you know of anything that can be done to improve this such as adjusting caching? Is it possible that because Kopia uploads small 20mb blobs that when the latency is within a few ms, it can fully saturate the upload, but when the latency is ~30 ms the uploads drastically slow down?
Any advise would be greatly appreciated. Happy to provide any logs or info if it is required. Hoping someone has had a similar experience to me.
This is just a shot in the dark but are you able to manually transfer a series of 20 MB/160 Mb test files between the endpoints in question for bench-marking? I use WebDAV so IDK if Kopia-Repo Server uses TLS. That may/may not be a variable but I’d be very surprised in this case if it were.
I know you’ve got some Iperf logs but it might be easier to take as many daemons out of the equation as possible for something that’s known to be fast in cleartext (eg: VSFTPd sans TLS vs SSH/SCP/SFTP). Iperf’s multitude of switches seem to really futz with expectations IMO (eg: reverse mode) but I can’t say I use it on a regular basis.
All this assumes all endpoints have matched/symmetrical pipes, of course.
Yeah, latency is always an issue, especially if you’re dealing with lots of transactions. I already notice a significant difference between a LAN and a Wifi snapshot performed by kopia. Trying to simulate this using iperf will be rather difficult, as iperf usually only measures the streaming performance.
However, this will only really matter for the first full snapshot as long as your dataset has no drastic changes between the following snapshots. What always worries me more than the snapshot speed is the restore speed. This is due to the fact that you will have to deal with the additional latencies that the remote storage will induce. So be prepared to take that into account and also check that out, if the restore speeds satisfies your needs.
After some further testing, it seems that most of my issue was with our network provider which is known for DDoS protection seems to be filtering some of the traffic on the Kopia repo server. After I moved to a Hetzner VM my average performance drastically increased. I am now able to push around 200-300mbps across the whole of North America.
After comparing the speeds it seems pretty much on par with BorgBase which was the service we were using before looking at moving to Kopia + B2.
I have started doing some testing on the restore speed now and that does seem to be about half the speed of BorgBase. If anyone has any suggestions on how to potentially optimize snapshot and restore speeds that would be great. From what I can see the ping from the client server to the kopia repository seems to make more of a difference than the ping from the kopia repo to the S3 bucket.
Any suggestions would be great though. I will continue testing different configs and update everyone once I find the most optimal setup. Our end goal is to have the quickest possible upload/download speeds to 1 single repo that manages 100+ client servers across the whole of NA.