Optimize kopia for Storj storage

General Topics
1

Hi

I’m using kopia with Storj (with S3 support).
Works very well, but I like to optimize.

Storj use 64MB per segment:

“The ideal part size for large files is 64MB, so that there is one Segment per part. Using a smaller Part size will result in a significant increase in the number of segments stored on Storj. At large scale, this could impact both the performance and cost of your storage”

How to configure kopia to use 64MB segment size?

Thanks in advance

roberto

2

You are using S3 gateway not native Storj API. Gateway will do, or will try to do what is the best with you having no control.

From S3 perspective you operate on files level. You can not influence what is happening behind the scene in this case.

3

Thanks for your reply

Seems I need to configure a custom segment size.

Look this:

for aws-cli command recommend this:

aws configure set default.s3.multipart_threshold 64MB

What is the default segment size on kopia backups ?

4

Please note that max file size kopia repo is using is about 20MB anyway. Multipart makes sense when uploading large files only.

Because of it multipart uploads are disabled:

Hence adding any multipart tweaking does not make any sense.

What you can do is to try to increase max pack size:

I also think that given info you provided Storj is not very good option as a general backup storage destination as it requires very specifically crafted files uploads. In simple terms it is only efficient for storing large files uploaded in 64MB chunks. But there are similarly priced or even cheaper S3 providers out there without such artificial limitations (myself I use iDrive e2) - at the end you have to decide what you use. In addition Storj with zero free egress (you will pay for running kopia maintenance when files need repacking) might end up relatively very expensive overall when used with kopia.

5

thanks for your reply @kapitainsky