# Upload and hash at the same time?

**URL:** <https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244>\
**Category:** Support\
**Created:** [June 30, 2022, 11:07pm UTC](https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244 "2022-06-30T23:07:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Qz4kngSbVkL8ZNpTGnrh](https://avatars.discourse-cdn.com/v4/letter/q/b487fb/32.png) [@Qz4kngSbVkL8ZNpTGnrh](https://kopia.discourse.group/u/Qz4kngSbVkL8ZNpTGnrh)\
**Post date:** [June 30, 2022, 11:07pm UTC](https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244/1 "2022-06-30T23:07:24Z")

</div>

When I run a backup, Kopia takes a minute to hash some files, then pause hashing when waiting for upload. The repository is using rclone.

Is there a way to allow them to run at the same time?

---

<div class="post-metadata">

**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\
**Post date:** [July 1, 2022, 5:00am UTC](https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244/2 "2022-07-01T05:00:41Z")

</div>

It’s mostly because rclone is slow. Hashing is pipelined with uploads but sometimes enough of upload queue builds up for hashing to stop.

---

<div class="post-metadata">

**Author:** ![Qz4kngSbVkL8ZNpTGnrh](https://avatars.discourse-cdn.com/v4/letter/q/b487fb/32.png) [@Qz4kngSbVkL8ZNpTGnrh](https://kopia.discourse.group/u/Qz4kngSbVkL8ZNpTGnrh)\
**Post date:** [July 3, 2022, 3:30am UTC](https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244/3 "2022-07-03T03:30:00Z")

</div>

I’ve taken a look at network graphs, and there appear to be small bursts of upload speed used (there would be one second of bandwidth used followed by several seconds of zero speed). Using rclone sync continuously saturates all available bandwidth. Is it due to the latency? If so, is there a way to initiate multiple uploads simultaneously to keep the data flowing? When using rclone sync, 8 parallel uploads occur simultaneously.

---

<div class="post-metadata">

**Author:** ![basldfalksjdf](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@basldfalksjdf](https://kopia.discourse.group/u/basldfalksjdf)\
**Post date:** [July 3, 2022, 4:12pm UTC](https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244/4 "2022-07-03T16:12:39Z")

</div>

The major problem here is that Rclone is basically acting as middleware allowing Kopia to connect to storage providers that Kopia does not support natively. Any time you have such middleware, there will always be at least some loss in performance, as Kopia needs to ensure that the files are being transferred correctly by Rclone. So you will never match the full speed that Kopia or Rclone have on their own. Still, perhaps it is possible for Kopia to run Rclone with different parameters that could speed up the process, but someone would need to submit a PR for that.

---

<div class="post-metadata">

**Author:** ![Qz4kngSbVkL8ZNpTGnrh](https://avatars.discourse-cdn.com/v4/letter/q/b487fb/32.png) [@Qz4kngSbVkL8ZNpTGnrh](https://kopia.discourse.group/u/Qz4kngSbVkL8ZNpTGnrh)\
**Post date:** [July 11, 2022, 8:24pm UTC](https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244/5 "2022-07-11T20:24:19Z")

</div>

> [@basldfalksjdf](#):
>
> Kopia needs to ensure that the files are being transferred correctly by Rclone

Sorry, I’m confused. What process is being performed by Kopia and Rclone?

I’ll try to take a look and see if the process can be optimized.

---

<div class="post-metadata">

**Author:** ![basldfalksjdf](https://avatars.discourse-cdn.com/v4/letter/b/ce73a5/32.png) [@basldfalksjdf](https://kopia.discourse.group/u/basldfalksjdf)\
**Post date:** [July 11, 2022, 10:04pm UTC](https://kopia.discourse.group/t/upload-and-hash-at-the-same-time/1244/6 "2022-07-11T22:04:06Z")

</div>

All I meant is that Kopia is essentially using Rclone as middleware, and that will always be slower than if Kopia was uploading directly. Granted, that does not mean that Kopia’s implementation of Rclone has been fully optimized. You can look at the rclone-specific code at [kopia/rclone\_storage.go at master · kopia/kopia · GitHub](https://github.com/kopia/kopia/blob/master/repo/blob/rclone/rclone_storage.go) to see what exactly is going on under the hood.
