Cache directories

Is there anywhere a definition of the different caches?

/kopia-cache/contents: 909 files 315.6 MB (limit 50 GB)
/kopia-cache/indexes: 14 files 36.7 MB
/kopia-cache/metadata: 234 files 1 GB (limit 1 GB)
/kopia-cache/own-writes: 6 files 614 B

Wondering if should increase metadata cache. If so, how one would determine what is a good number?

Use case is like this:

  • A couple of machines rsync to a centralized storage machine
  • Run kopia against the synced folders on the storage machine

content cache - stores file contents (from p blobs) - one file per content

indexes cache - stores indexes (n,m and l blobs) downloaded from blob storage

metadata cache - stores metadata (directory listings, snapshot manifests, from q blobs) - the entire blob is cached together, since metadata is frequently accessed together

own-writes - is a short-lived cache (~1-2 hours) that stores index files (n, m, l blobs) before they get uploaded to the server, in case server supports weak eventual consistency

For performance reasons you want your metadata cache size to be large enough so that all metadata fits in it.

You may want to run kopia blob stats --prefix=q and set metadata size to 2x-5x total size to accommodate for future growth.Good rule of thumb is few gigabytes if you can spare it.

Content cache is not critical as it’s only used for browsing/mounting/restores. Since you rarely need to access the same file more than once/few times in these situations. When in doubt, give it ~10 GB, although probably much less will suffice.

Index and own writes caches are not configurable, since they must hold all indexes downloaded from the server to be able to memory-map them for quick access.

2 Likes