I’ve been using Kopia with Storj as the storage solution for quite a while now. I’ve backed up large amounts of data with many snapshots over the course of over a year. It has always worked fine until a few days ago when it suddenly stopped connecting. At first I thought it was just a random hangup but Kopia kept failing to connect to the repository despite several restarts of both Kopia and my PC, and several attempts at reconnecting to the repository. So I’ve checked the logs and found the (suspected) reason there: Apparently, the large repository is causing the request to hit some sort of API or gateway limit in the background (see screenshot below). The error message reads:
“refresh attempt failed: error refreshing: error loading write epoch: error listing all blobs: ListObjects(V2): listing too many items for gateway-side filtering using arbitrary delimiter/prefix”
Since I thought it might be some API compatibility issue, I’ve even tried downgrading Kopia from 0.20.1 to 0.18.2 but it hasn’t helped.
I’d really appreciate any help with this since I’m currently unable to recover any of the backups. If this isn’t fixable, I’m ok with setting up a new backup repository, however, it would be nice to at least be able to copy over some of the data.
Does not sound like kopia problem but Storj limitation. I guess their S3 gateway has some limits. Ask on their forums - maybe such limits can be increased.
You can always use rclone to copy/move all your repo to some other more reliable provider.
Thanks for your response! Sure, it seems to hit some limit with the Storj service, however, since this has happened rather suddenly, I doubt that it’s a general problem with Storj. Is it possible that one of the latest updates to Kopia has changed something in the way it requests information from the storage service? I find the specific reference to filtering and “arbitrary delimiter/prefix” strange since it has never appeared before.
EDIT: I’ve just tried listing all files via Rclone and it works without a problem. So this doesn’t seem to be an inherent limitation of the gateway but the function used to request information from the API (i.e. ListObjects V2).
I assume you are using Storj S3 access. If it does not comply fully with S3 specification then I do not think it is the right path to tweak kopia to work with some non compliant providers…
So yes it is possible that something changed in kopia. But at the same time it does not mean that kopia should even try to accommodate situation when some S3 issues are exposed with specific vendor.
I think if it was general S3 issue we would see more similar error reports.
It turns out that apparently Kopia doesn’t use / as a delimiter or does not end prefixes with / which leads to a less efficient list/fetch operation. If this is really the case, it might be an easy fix.
Regardless of that, does anyone have an idea how to avoid this problem or how to at least circumvent it to get the repository working again?
I understand that this may not be an issue occurring with many other storage backends but if the “incompatibility” is really just a setting in the list/fetch operation, it would be great if this could be addressed in a future update. Because as is, any repository reaching a size of around 2 TB can easily reach beyond 100,000 objects and cause the problem described above.
For context:
From what I understand, the issue seems to originate not from a lack of S3 compliance but from the fact that Storj is a distributed, zero-trust encrypted storage. As a result, even paths are encrypted and need to be decrypted when listed. That can make listing operations slower than on other platforms (even though in my experience it is very fast when using rclone or their native uplink CLI tool).
The particular problem occurring here seems to originate from the fact that “unterminated prefixes” (i.e., prefixes not ending with a delimiter) trigger an exhaustive listing of all objects. And once there are more than 100,000 objects, the S3 gateway seems to cause the error described above. I might be wrong but it seems this limit is circumvented by simply always adding a “/” at the end of each object or directory (prefix) listing.
Interestingly, when doing the same operation through rclone (just listing all objects in the bucket), it works and is quite fast, listing all objects within a few seconds.
Question about the alternative:
If none of these issues occur when using rclone, an alternative may be to just switch from S3 to rclone as the backend connection. Therefore, my question: Is it possible to migrate or convert a Kopia S3 backend repository to a Kopia rclone backend repository? Or are these two structured differently?
Regarding your investigation and conclusions you could create an issue on github hoping that somebody will “fix” it one day. But I would not keep my breath for it:) Unless you are happy to DYI and submit PR.
It should work without any issue out of the box but keep in mind that you are adding extra “moving parts” making all solution more prone to problems. If you go this path you could also ditch all S3 gateway as rclone supports native Storj connectivity (but read about caveats in rclone docs).
