Slow restores locally from filesystem

Support
1

The context is I use Kopia to snapshot a raid array onto an external drive in a USB enclosure.

I’ve not tried a full restore before but I’m using the snapshots/kopia to migrate data by running kopia server on another host and connecting to the repository store on the external drive having plugged it into the same host.

So USB drive plugged into host, running Kopia all local and no network connectivity to cause any troubles.

I’ve then been selecting the most recent snapshot and restoring that to a new array by using the webui and just leaving the default options.

However this results in really quite slow speeds (2.7 TB in 18h30m so roughly 40MB/s?)

I’ve checked a hdparm result reading from the external disk and it shows something well over double that speed on reads.

So I’ve ran an alternative restore by mounting the snapshot as a volume and using rsync and that then produces a speed when copying of around 80MB/s. Which while nothing to shout home about is double the result with Kopia alone and obviously cuts the restore time down significantly.

I’m not really trying to solve a problem as such (rsync is doing me fine now) but I’m wondering if I’m missing something in configuring Kopia or the restore?

2

That’s to be expected, since Kopia chunks files for dedup and that can cause a significant number of files to access for restoring one source file. As it is, latency of the drive is mainly the resulting “transfer rate”.

To add to the issue, maintenance will cause a degree of fragmentation on the drive the repo is located on. There’s no chance, that you will ever achieve the same throughput as in a regular file transfer or copy operation.

1 Like
3

Thanks interesting to know for future, I’ll have to stick to mounting the snapshot and rsync for any future large scale restoration :+1:

4

It won’t be any faster as behind mount is the same mechanism of reconstructing files from repository. Plus I think that kopia mount is single threaded - so it will be even slower. Mount is only good to poke around your backup and restore some random file or two if needed.

You can not have all. Thx to kopia you can squeeze a lot of data into given space volume (deduplication and compression) but it comes at the cost of speed.

If you need speed then you need different tools but also much more space so files can be transferred without any additional operations.