# Optimize kopia for Storj storage **URL:** \ **Category:** General Topics\ **Created:** [May 4, 2024, 1:54pm UTC](https://kopia.discourse.group/t/optimize-kopia-for-storj-storage/2880 "2024-05-04T13:54:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![rjp](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/rjp/32/1030_2.png) [@rjp](https://kopia.discourse.group/u/rjp)\ **Post date:** [May 4, 2024, 1:54pm UTC](https://kopia.discourse.group/t/optimize-kopia-for-storj-storage/2880/1 "2024-05-04T13:54:47Z")
Hi I’m using kopia with Storj (with S3 support). Works very well, but I like to optimize. Storj use 64MB per segment: > **[Optimizing Multipart Upload Part Size for Better Performance - Storj Docs](https://docs.storj.io/dcs/api/s3/multipart-upload/multipart-part-size)** > > Guidance on optimizing multipart upload part size to improve Storj performance and control costs. Recommended size is 64MB. “The ideal part size for large files is 64MB, so that there is one Segment per part. Using a smaller Part size will result in a significant increase in the number of segments stored on Storj. At large scale, this could impact both the performance and cost of your storage” How to configure kopia to use 64MB segment size? Thanks in advance roberto ---
**Author:** ![kapitainsky](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/kapitainsky/32/535_2.png) [@kapitainsky](https://kopia.discourse.group/u/kapitainsky)\ **Post date:** [May 4, 2024, 3:19pm UTC](https://kopia.discourse.group/t/optimize-kopia-for-storj-storage/2880/2 "2024-05-04T15:19:12Z")
You are using S3 gateway not native Storj API. Gateway will do, or will try to do what is the best with you having no control. From S3 perspective you operate on files level. You can not influence what is happening behind the scene in this case. ---
**Author:** ![rjp](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/rjp/32/1030_2.png) [@rjp](https://kopia.discourse.group/u/rjp)\ **Post date:** [May 4, 2024, 3:24pm UTC](https://kopia.discourse.group/t/optimize-kopia-for-storj-storage/2880/3 "2024-05-04T15:24:07Z")
Thanks for your reply Seems I need to configure a custom segment size. Look this: > **[Optimizing Multipart Upload Part Size for Better Performance - Storj Docs](https://docs.storj.io/dcs/api/s3/multipart-upload/multipart-part-size)** > > Guidance on optimizing multipart upload part size to improve Storj performance and control costs. Recommended size is 64MB. for aws-cli command recommend this: **aws configure set default.s3.multipart\_threshold 64MB** What is the default segment size on kopia backups ? ---
**Author:** ![kapitainsky](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/kapitainsky/32/535_2.png) [@kapitainsky](https://kopia.discourse.group/u/kapitainsky)\ **Post date:** [May 5, 2024, 5:34am UTC](https://kopia.discourse.group/t/optimize-kopia-for-storj-storage/2880/4 "2024-05-05T05:34:09Z")
Please note that max file size kopia repo is using is about 20MB anyway. Multipart makes sense when uploading large files only. Because of it multipart uploads are disabled: > Hence adding any multipart tweaking does not make any sense. What you can do is to try to increase max pack size: > > > Is it possible to specify a custom blob size? > > For instance, borg can crea…te ~500MB files which means the repo is made of fewer files (useful for providers like Google Team Drive that limit the number of files, but not the total space) > > \`\`\` > \-rw------- 1 root root 501M 16 lug 04.00 1219 > \-rw------- 1 root root 507M 16 lug 04.00 1220 > \-rw------- 1 root root 502M 16 lug 04.00 1221 > \-rw------- 1 root root 501M 16 lug 04.00 1222 > \-rw------- 1 root root 283M 16 lug 04.01 1223 > \`\`\` > > Kopia instead for the same repo creates ~20MB files (with default options): > > \`\`\` > rclone ncdu v1.59.0 - use the arrow keys to navigate, press ? for help > \-- REMOTE:/HOST/kopia/REPO/p22 ------------------------------------------------------------------------------------------ > 20.287Mi \[##########\] /c4e > \`\`\` > > I don't know if tweaking sharding is the way to go because if I got it right, changing the sharding parameter still maintain the same number of ~20MB shards, it just changes the amount of shards per folder (getting more folders and shorter blob file names) I also think that given info you provided Storj is not very good option as a general backup storage destination as it requires very specifically crafted files uploads. In simple terms it is only efficient for storing large files uploaded in 64MB chunks. But there are similarly priced or even cheaper S3 providers out there without such artificial limitations (myself I use [iDrive e2](https://www.idrive.com/s3-storage-e2/pricing)) - at the end you have to decide what you use. In addition Storj with zero free egress (you will pay for running kopia [maintenance](https://kopia.io/docs/advanced/maintenance/) when files need repacking) might end up relatively very expensive overall when used with kopia. ---
**Author:** ![rjp](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/rjp/32/1030_2.png) [@rjp](https://kopia.discourse.group/u/rjp)\ **Post date:** [May 11, 2024, 4:53pm UTC](https://kopia.discourse.group/t/optimize-kopia-for-storj-storage/2880/5 "2024-05-11T16:53:28Z")
thanks for your reply @kapitainsky ---
**Author:** ![peter](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/peter/32/441_2.png) [@peter](https://kopia.discourse.group/u/peter)\ **Post date:** [March 15, 2026, 6:11pm UTC](https://kopia.discourse.group/t/optimize-kopia-for-storj-storage/2880/6 "2026-03-15T18:11:43Z")
When you’re using kopia with Storj via S3 you should be aware that the S3 gateway has a limitation with the number of blobs within a directory. When your repo is small enough, everything is fine. Mine just grew over that thrashold and I cannot access it anymore via S3. … Just to be aware. > > > I attempted to switch to Storj (\[Globally Distributed Cloud Object Storage\](http…s://www.storj.io/)) as a blobstore. This doesn't currently work very well. Two observations about this: > > \* \`kopia repository sync-to rclone\` seems to work but is very slow > \* the Storj S3 compatibility interface works just fine for \*small\* blobstores but is not adequate for production use because the Storj S3 gateway limits \`s3\_storage.go:/ListBlobs()\` to a thousand blobs (per documentation) total if using prefix matching. > > I see two solutions: > > \* run a local Storj S3 gateway with much larger limits > \* write a "native" Storj blob implementation in Kopia. > > I would expect that the "native" implementation (i.e. Storj blob implementation) is preferable for its performance and robustness. > > @jkowalski et al: are you open to PRs implementing such a feature? If so, I will attempt to write this.