# Repo folder structure suggestion | default .shards option suggestion

**URL:** <https://kopia.discourse.group/t/repo-folder-structure-suggestion-default-shards-option-suggestion/2622>\
**Category:** Feature Requests\
**Created:** [February 1, 2024, 12:34pm UTC](https://kopia.discourse.group/t/repo-folder-structure-suggestion-default-shards-option-suggestion/2622 "2024-02-01T12:34:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![9-2-1](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/9-2-1/32/931_2.png) [@9-2-1](https://kopia.discourse.group/u/9-2-1)\
**Post date:** [February 1, 2024, 12:34pm UTC](https://kopia.discourse.group/t/repo-folder-structure-suggestion-default-shards-option-suggestion/2622/1 "2024-02-01T12:34:00Z")

</div>

The default [sharding](https://kopia.io/docs/advanced/sharding/) option of kopia and kopiaUI are different: kopia use [1, 3] while kopiaUI use [3, 3].

The problem is that it will create lots of folders for s-files and x-files (session files and index files), and nearly every actions will list these folders, while listing many folder on a HDD or network drive is very slow. (using `--flat` option will put many files in one folder, which is slow too)

I think the default `.shards` can be:

```JSON
{
    "default": [1, 2],
    "maxNonShardedLength": 20,
    "overrides": [
        { "prefix": "q", "shards": [1, 1] },
        { "prefix": "s", "shards": [1] },
        { "prefix": "x", "shards": [1] }
    ]
}

```

The folder structure will be:

```auto
p\ # data blocks
    04\
    25\
    a6\
    ...\
q\ # metadata blocks
    1\
    3\
    a\
    ...\
s\ # session files
    ****.f
x\ # index blocks
    n23_ ****.f
    n24_ ****.f
    r0_7_ ****.f
    r8_15_ ****.f
    s0_ ****.f
    s1_ ****.f
    ...
    s22_ ****.f
kopia_ ****.f
xw ****.f
xe23.f
xe24.f

```

The sharding strategy is based on some assumptions:

1. More files and folders in a folder, the folder is slower to read file and to create new files.
2. Files are faster to be listed in a single folder then in many different folders.
3. P-files and q-files are only listed when running maintenance, otherwise they are directly created or read (with the help of indexes), so these files can be grouped into folders to boost speed.
4. X-files are listed on every operation so they are in a single folder.
5. While s-files are only listed when running maintenance, they are created and deleted on every operation and the number of them is too small. So it is better to place these files in a single folder instead of creating many folder then empty them, so these folders will not be created and listed.
6. Maintenance is running regularly, so the numbers of x-files and q-files are small.
7. a ~400GB repository contains about 40000 p-files and 200 q-files, so grouping these into 256(folders)\*150(files) and 16(folders)\*13(files) is better then using 4096 folders.

* * *

Here is the same issue I have created on github:

> <https://github.com/kopia/kopia/issues/3602>
>
> The default \[sharding\](https://kopia.io/docs/advanced/sharding/) option of kopia… and kopiaUI are different: kopia use \[1, 3\] while kopiaUI use \[3, 3\].
> 
> The problem is that it will create lots of folders for s-files and x-files (session files and index files), and nearly every actions will list these folders, and listing many folder on a HDD or network drive is very slow.
> 
> I think the default \`.shards\` can be:
> \`\`\` JSON
> {
> "default": \[1, 2\],
> "maxNonShardedLength": 20,
> "overrides": \[
> { "prefix": "q", "shards": \[1, 1\] },
> { "prefix": "s", "shards": \[1\] },
> { "prefix": "x", "shards": \[1\] }
> \]
> }
> \`\`\`
> The folder structure will be:
> \`\`\`
> p\\ # data blocks
> 04\\
> 25\\
> a6\\
> ...\\
> q\\ # metadata blocks
> 1\\
> 3\\
> a\\
> ...\\
> s\\ # session files
> \*\*\*\*.f
> x\\ # index blocks
> n23\_\*\*\*\*.f
> n24\_\*\*\*\*.f
> r0\_7\_\*\*\*\*.f
> r8\_15\_\*\*\*\*.f
> s0\_\*\*\*\*.f
> s1\_\*\*\*\*.f
> ...
> s22\_\*\*\*\*.f
> kopia\_\*\*\*\*.f
> xw\*\*\*\*.f
> xe23.f
> xe24.f
> \`\`\`
> The sharding strategy is based on some assumptions:
> 1. More files and folders in a folder, the folder is slower to read file and to create new files.
> 2. Files are faster to be listed in a single folder then in many different folders.
> 3. P-files and q-files are only listed when running maintenance, otherwise they are directly created or read (with the help of indexes), so these files can be grouped into folders to boost speed.
> 4. X-files are listed on every operation so they are in a single folder.
> 5. While s-files are only listed when running maintenance, they are created and deleted on every operation and the number of them is too small. So it is better to place these files in a single folder instead of creating many folder then empty them, so these folders will not be created and listed.
> 6. Maintenance is running regularly, so the numbers of x-files and q-files are small.
> 7. a ~400GB repository contains about 40000 p-files and 200 q-files, so grouping these into 256(folders)\*150(files) and 16(folders)\*13(files) is better then using 4096 folders.
> 
> | Prefix | Description | Listing frequently? | Large numbers of file? | Strategy |
> | - | - | - | - | - |
> | P | Data pack | No | Yes | Group by first two chars |
> | Q | Metadata pack | No | No | Group by first char |
> | S | Session file (see \[design document\](https://github.com/kopia/kopia/issues/731)) | No | No | All in one folder |
> | X | Index file | Yes | No | All in one folder |

* * *

After explored the forum, I found a hidden command to modify a local repository to the shards format I have mentioned:

```bash
kopia blob shards modify --path=/path/to/repo --i-am-sure-kopia-is-not-running --default-shards=1,2 --override=q=1,1 --override=x=1 --override=s=1 --unsharded-length 20

```

It will move these files to their new places. It may take a long time, while you cannot use the repo before it done.

---

<div class="post-metadata">

**Author:** ![9-2-1](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/9-2-1/32/931_2.png) [@9-2-1](https://kopia.discourse.group/u/9-2-1)\
**Post date:** [February 1, 2024, 12:48pm UTC](https://kopia.discourse.group/t/repo-folder-structure-suggestion-default-shards-option-suggestion/2622/2 "2024-02-01T12:48:22Z")

</div>

This `.shards` file group these `_log_` files in one single folder (`_log_\ ***.f` insteads of `_\lo\g_***.f`),:

```JSON
{
    "default": [1, 2],
    "maxNonShardedLength": 20,
    "overrides": [
        { "prefix": "_log_", "shards": [5] },
        { "prefix": "q", "shards": [1, 1] },
        { "prefix": "s", "shards": [1] },
        { "prefix": "x", "shards": [1] }
    ]
}

```

Modify command:

```bash
kopia blob shards modify --path=/path/to/repo --i-am-sure-kopia-is-not-running --default-shards=1,2 --override=_log_=5 --override=q=1,1 --override=x=1 --override=s=1 --unsharded-length 20

```

| Prefix | Description | Listing frequently? | Large numbers of file? | Strategy |
| --- | --- | --- | --- | --- |
| P | Data pack | No | Yes | Group by first two chars |
| Q | Metadata pack | No | No | Group by first char |
| S | Session file (see [design document](https://github.com/kopia/kopia/issues/731)) | No | No | All in one folder |
| X | Index file | Yes | No | All in one folder |
| \_log\_ | log file | No | No | All in one folder |