# Low read performance per distinct hashing process **URL:** \ **Category:** Support\ **Created:** [September 6, 2020, 4:21pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90 "2020-09-06T16:21:03Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 9, 2020, 7:04am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/21 "2020-09-09T07:04:53Z")
> [@jkowalski](#): > > BTW. I finally have a fix for the painfully slow checkpointing. The fix avoids having to do restarts completely. šŸ‘šŸ» Canā€™t wait to try that out - when will we be able to test that? ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 9, 2020, 12:31pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/22 "2020-09-09T12:31:55Z")
You can build from the PR branch if youā€™re really curious. But given this has probably some new bugs I would recommend waiting (that is why I test on clean repository) ---
**Author:** ![julio](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/julio/32/36_2.png) [@julio](https://kopia.discourse.group/u/julio)\ **Post date:** [September 11, 2020, 12:04am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/23 "2020-09-11T00:04:16Z")
@francisco The results for the splitter benchmark are ranked by execution time starting starting with the best (lowest execution time) at the top. The difference between some of them is minimal, thus within some error margin they are equivalent. And different executions on the same inputs and machine may provide slightly different ranks. The columns are: - Rank / order - Splitter name - Total Execution time, lower is better - Number of resulting chunks for a fixed / constant input size. - Distribution of the chunk sizes. (minimum, maximum chunk sizes and in between). I hope this helps. ---
**Author:** ![julio](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/julio/32/36_2.png) [@julio](https://kopia.discourse.group/u/julio)\ **Post date:** [September 11, 2020, 12:11am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/24 "2020-09-11T00:11:00Z")
> [@francisco](#): > > Canā€™t find a way either. I guess will have to create / test / migrate repo Correct. - These are repository-wide parameters (splitter, hashing and encryption) that are set at creation time. - The solution is to migrate the repository as you point out. If these parameter were to be dynamically changed, it would involve rewriting all the contents in the repository, which is equivalent to a migration. ---
**Author:** ![julio](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/julio/32/36_2.png) [@julio](https://kopia.discourse.group/u/julio)\ **Post date:** [September 11, 2020, 12:18am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/25 "2020-09-11T00:18:10Z")
The 35 M/s throughput cap due to the splitter is surprising. It seems to indicate that CPU may be the bottleneck here. The expectation is that I/O (reads for the most part in this case) would be the bottleneck, but that depends on the available CPU, memory and I/O resources. @budy May I ask what your setup is? Also, are you running kopia in ā€œserverā€ mode? ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 11, 2020, 7:59am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/26 "2020-09-11T07:59:20Z")
Hi @julio, I am testing kopia in a client-server setup, where the clients are Supermicro servers with these two CPUs: E5-2670 v2 @ 2.50GHz (10 cores) and E5-2660 v3 @ 2.60GHz (10 Cores). The latter being in a newer server node. Those are the clients which run kopia. The storage node, which is another server, runs kopia in server mode. When running on the older Xeon, I am achieving approx. 35 MB/s, on the newer Xeon 47 MB/s when hashing/splitting a single large file. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 11, 2020, 10:25pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/27 "2020-09-11T22:25:02Z")
See [https://github.com/kopia/kopia/pull/606](https://github.com/kopia/kopia/pull/606) for a PR that improves the single-core throughput of splitters. In my tests Iā€™m seeing throughput increase of about 40% for BUZHASH ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 12, 2020, 3:19am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/28 "2020-09-12T03:19:48Z")
BTW, this has been merged and is available in `unstable` via RPM and APT repository. ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 12, 2020, 8:19am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/29 "2020-09-12T08:19:27Z")
Oh great! Now, I will give it a spin and report back. Thanks! ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 12, 2020, 12:21pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/30 "2020-09-12T12:21:16Z")
Yesssā€¦ this looks much better alreadyā€¦ ``` Snapshotting root@pandora:/mnt/pve/vmBackup ... \ 40 hashing, 117 hashed (1.1 TB), 0 cached (0 B), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... | 40 hashing, 55 hashed (1.5 TB), 117 cached (237.8 GB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... | 40 hashing, 43 hashed (1.6 TB), 172 cached (645.5 GB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... | 40 hashing, 27 hashed (1.6 TB), 215 cached (883.3 GB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... / 40 hashing, 2 hashed (1.5 TB), 242 cached (1 TB), 0 uploaded (0 B), 0 errors rs ``` The overalll throughput also increased, although I donā€™t have any exact numbers for a single process. I will see to that after the first snapshot has been completely taken. ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 12, 2020, 4:21pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/31 "2020-09-12T16:21:14Z")
Wellā€¦ here we are againā€¦ just after I went away things started to stallā€¦ ``` Snapshotting root@pandora:/mnt/pve/vmBackup ... \ 40 hashing, 117 hashed (1.1 TB), 0 cached (0 B), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... | 40 hashing, 55 hashed (1.5 TB), 117 cached (237.8 GB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... | 40 hashing, 43 hashed (1.6 TB), 172 cached (645.5 GB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... | 40 hashing, 27 hashed (1.6 TB), 215 cached (883.3 GB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... \ 40 hashing, 2 hashed (1.7 TB), 242 cached (1 TB), 0 uploaded (0 B), 0 errors rs ! Saving a checkpoint... \ 40 hashing, 0 hashed (1.7 TB), 244 cached (1 TB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... / 24 hashing, 16 hashed (1.7 TB), 244 cached (1 TB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... / 40 hashing, 0 hashed (1.7 TB), 244 cached (1 TB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... - 40 hashing, 0 hashed (1.7 TB), 244 cached (1 TB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... - 40 hashing, 0 hashed (1.7 TB), 244 cached (1 TB), 0 uploaded (0 B), 0 errors ! Saving a checkpoint... | 40 hashing, 0 hashed (494.3 GB), 244 cached (1 TB), 0 uploaded (0 B), 0 errors ``` ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 12, 2020, 4:36pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/32 "2020-09-12T16:36:59Z")
If you want to try building Kopia from `jkowalski/kopia ` using `checkpointing-fix` branch, you should see this fixed. The PR is not merged yet, but Iā€™d appreciate folks taking it for a spin. ```auto $ git clone https://github.com/jkowalski/kopia $ cd kopia $ git checkout checkpointing-fix $ go install $ ~/go/bin/kopia snapshot create ... ``` (Btw this also has splitter performance fix now merged). ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 12, 2020, 4:48pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/33 "2020-09-12T16:48:58Z")
Arghā€¦ silly meā€¦ I totally ignored that the checkpointing fix has not yet been committedā€¦ However a quick check of a snapshotting a single file revealed more than a 40% increase. As far as I could check the performance increase was more in the range of 2x, probably even more. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 13, 2020, 3:01pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/34 "2020-09-13T15:01:17Z")
Itā€™s merged now. Can you update Kopia and give it another try? ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 14, 2020, 2:39am UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/35 "2020-09-14T02:39:27Z")
The fixes are now available in v0.7.0-rc1 release. Can you let me know whether the performance has improved for you? ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 14, 2020, 4:57pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/36 "2020-09-14T16:57:31Z")
Ehā€¦ I am on a short break until thursday and I wonā€˜t have access to my gear before. Sorryā€¦ ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 15, 2020, 3:31pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/37 "2020-09-15T15:31:23Z")
Okay, I couldnā€˜t resist and updated remotely. remotely. Preparing to unpack ā€¦/kopia\_20200913.0.184705\_amd64.deb ā€¦ Unpacking kopia (20200913.0.184705) over (20200911.0.201201) ā€¦ Setting up kopia (20200913.0.184705) ā€¦ Then I created a snapshot and now it seems, that Kopia doesnā€˜t perform any checkpoints at all, although I didnā€˜t specify any checkpoint interval. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [September 15, 2020, 4:01pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/38 "2020-09-15T16:01:21Z")
It does checkpoint internally, I removed displaying that since it wasnā€™t adding much value. ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 15, 2020, 4:03pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/39 "2020-09-15T16:03:03Z")
Thought so. Looks good to me! ---
**Author:** ![budy](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/budy/32/38_2.png) [@budy](https://kopia.discourse.group/u/budy)\ **Post date:** [September 15, 2020, 7:55pm UTC](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90/41 "2020-09-15T19:55:46Z")
Works great now: ``` root@poseidon:~# kopia snapshot /mnt/pve/vmBackupSnapshotting root@pandora:/mnt/pve/vmBackup ... * 0 hashing, 332 hashed (7.2 TB), 60 cached (32.3 GB), 0 uploaded (0 B), 0 errors 73.4% Created snapshot with root kee5e98d8ea2b7a6d31848fe58143c0a1 and ID f3b2abd5b4ab3b2037d1c9fdd3fafd64 in 5h8m7s root@poseidon:~# kopia snapshot list root@pandora:/mnt/pve/vmBackup 2020-09-15 16:14:24 CEST kee5e98d8ea2b7a6d31848fe58143c0a1 7.3 TB drwxr-xr-x files:392 dirs:23 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1) ``` [Previous page](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90.md?page=1) [Next page](https://kopia.discourse.group/t/low-read-performance-per-distinct-hashing-process/90.md?page=3)