Help needed sharding

Support
1

I’d like to setup sharding in an S3 repository. First I put a .shards file into the bucket, but it is ignored.
Where does it have to go to? Sharding | Kopia says: ““.shards” file located in every repository”. I thought, this must be the S3 bucket in my case.
The kopia blob shards modify --path= command expects a “sharded directory path”. Seemingly a local one. At least it looks locally at the given path for the “.shards” file.
How does a S3 “directory path” have to look like?
Or more general: how can I do sharding inside my bucket?
Reason is observing the sizes of the kopia files, which is easier in the AWS console to summarize by blob type, if the different blob types are inside directories.

2

Sharding is only for local-like (filesystem sftp, webdav, rclone, etc.) repositories and generally only helps performance. Cloud buckets can scale horizontally so it’s not needed (nor supported).

What exactly are you trying to achieve?

3

To explain further: To be able to make a good configuration for the storage class of different file types and in general observe the storage.
For instance, for 170 pages of p blobs per Terrabyte, it’s almost impossbile to summarize the space needed for all of them, if they’re not inside an own folder.

4

Perhaps kopia blob stats --prefix=p can help here?

5

Sorry, forgot: Also to move blobs to a different storage class, in case I decide differently than when uploaded. Or to delete or to “restore request” all or many of them at once. All this is easy and straightforward inside the AWS console, if it wheren’t 1000s of files, that need to be handled.
I’ll look into blob stats anyway, thanks.

6

As I posted here, doesn’t this mean that a local, sharded repository cannot be rclone’d to cloud storage, as the latter has no support for sharding? Having different repository format requirements for local and cloud-based storage seems quite bad.

This is apart from the cosmetic issue – consider a repository produced by restic:

- config
- data/
  - 00/
  - 01/
  - 02/
  - ....
- index/
- keys/
- locks/
- snapshots/

Tidy and organized, inspires confidence that the repository format is solid and well thought-out. Kopia is just a huge list of files.