Hi,
I am currently evaluating Kopia and trying to set it up with a server (via rclone), that seems to have some form of rate limitation. When there are too many connections within an unknown timespan, new ones are simply declined. This lock only persists for a minute or two.
In this time, Kopia retries the action (which is good!), but it gives up after 10 tries within only 6 seconds.
2024-10-31T17:02:15.962427Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-1874816528590208550: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#0), sleeping for 100ms before retrying
2024-10-31T17:02:23.437294Z DEBUG retry got error MOVE p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-5702604869473354346: 403 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#1), sleeping for 150ms before retrying
2024-10-31T17:02:23.653201Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-3780161810903662686: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#2), sleeping for 225ms before retrying
2024-10-31T17:02:23.955023Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-7953155419298284000: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#3), sleeping for 337.5ms before retrying
2024-10-31T17:02:24.356233Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-5652531382805946283: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#4), sleeping for 506.25ms before retrying
2024-10-31T17:02:24.930018Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-242526806045731424: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#5), sleeping for 759.375ms before retrying
2024-10-31T17:02:25.785368Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-551731680282466621: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#6), sleeping for 1.1390625s before retrying
2024-10-31T17:02:26.998709Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-6895356586295339808: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#7), sleeping for 1.70859375s before retrying
2024-10-31T17:02:37.630353Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-5165351217291380670: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#8), sleeping for 2.562890625s before retrying
2024-10-31T17:02:40.283446Z DEBUG retry got error Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-4381490030233784767: 405 when PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) (#9), sleeping for 3.844335937s before retrying
2024-10-31T17:02:44.131547Z DEBUG kopia/repo [STORAGE] PutBlob {"blobID":"pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e","length":23699858,"error":"unable to complete PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) despite 10 retries: Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-4381490030233784767: 405","errorVerbose":"Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-4381490030233784767: 405\nunable to complete PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) despite 10 retries\ngithub.com/kopia/kopia/internal/retry.internalRetry[...]\n\tD:/a/kopia/kopia/internal/retry/retry.go:94\ngithub.com/kopia/kopia/internal/retry.WithExponentialBackoff[...]\n\tD:/a/kopia/kopia/internal/retry/retry.go:31\ngithub.com/kopia/kopia/internal/retry.WithExponentialBackoffNoValue\n\tD:/a/kopia/kopia/internal/retry/retry.go:100\ngithub.com/kopia/kopia/repo/blob/retrying.retryingStorage.PutBlob\n\tD:/a/kopia/kopia/repo/blob/retrying/retrying_storage.go:34\ngithub.com/kopia/kopia/repo/blob/logging.(*loggingStorage).PutBlob\n\tD:/a/kopia/kopia/repo/blob/logging/logging_storage.go:119\ngithub.com/kopia/kopia/repo/blob/storagemetrics.(*blobMetrics).PutBlob\n\tD:/a/kopia/kopia/repo/blob/storagemetrics/storage_metrics.go:100\ngithub.com/kopia/kopia/repo/blob/throttling.(*throttlingStorage).PutBlob\n\tD:/a/kopia/kopia/repo/blob/throttling/throttling_storage.go:98\ngithub.com/kopia/kopia/repo/blob/beforeop.beforeOp.PutBlob\n\tD:/a/kopia/kopia/repo/blob/beforeop/beforeop.go:50\ngithub.com/kopia/kopia/repo/content.(*SharedManager).writePackFileNotLocked\n\tD:/a/kopia/kopia/repo/content/content_manager_lock_free.go:194\ngithub.com/kopia/kopia/repo/content.(*SharedManager).prepareAndWritePackInternal\n\tD:/a/kopia/kopia/repo/content/content_manager.go:565\ngithub.com/kopia/kopia/repo/content.(*WriteManager).writePackAndAddToIndexUnlocked\n\tD:/a/kopia/kopia/repo/content/content_manager.go:513\ngithub.com/kopia/kopia/repo/content.(*WriteManager).addToPackUnlocked\n\tD:/a/kopia/kopia/repo/content/content_manager.go:341\ngithub.com/kopia/kopia/repo/content.(*WriteManager).WriteContent\n\tD:/a/kopia/kopia/repo/content/content_manager.go:838\ngithub.com/kopia/kopia/repo/object.(*objectWriter).prepareAndWriteContentChunk\n\tD:/a/kopia/kopia/repo/object/object_writer.go:206\ngithub.com/kopia/kopia/repo/object.(*objectWriter).flushBuffer.func1\n\tD:/a/kopia/kopia/repo/object/object_writer.go:173\nruntime.goexit\n\tC:/hostedtoolcache/windows/go/1.22.2/x64/src/runtime/asm_amd64.s:1695","duration":"35.0846182s"}
2024-10-31T17:02:44.134616Z ERROR object async write error: unable to write content chunk 0 of FILE:ABC.JPG: unable to write pack: error writing pack: can't save pack data blob pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e: error writing pack file: unable to complete PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) despite 10 retries: Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-4381490030233784767: 405: unable to write pack: error writing pack: can't save pack data blob pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e: error writing pack file: unable to complete PutBlob(pc893be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e) despite 10 retries: Write p/c89/3be85af32385673ee733e60f25daa-s18ac9b88e71bfcf212e.f-4381490030233784767: 405
Is there a way to make the retry count and timeout somewhat configurable via command line or other configuration? It currently looks hardcoded to me.
PS: I tried limiting the number of requests in rclone to 1 per second, which reduces the chance for this rate limit to kick in, but it still happens + it made everything extremely slow. Chosing another remote server is no option for me.