Optimize Kopia cache usage when all repository content data are available locally

Support
1

Hi,

Setup:

I have a kopia repository of size 100Gb to 1Tb on the cloud. This repository is fully synced (via kopia sync…) to a local hard disk.

100% of the cloud repository data is available locally.

Action:

Kopia snapshot verify runs and via fatrace I observed the « /path/to/repo/cache/contents » contains about 11G of contents and constantly being written to.

Expected behaviour:

When all repo data are available locally, I was not expecting kopia snapshot verify to write anything in the content cache. Decrypted/uncompressed Content blobs needed to verify a repo file should be kept in RAM only.

Questions

  1. Why is kopia using the « path/to/repo/cache/contents » folder in this context/setup ?

  2. I would like to eliminate as much as possible disk writes to the cache (data are stored on SMR platter disks connected over USB 3.0) and it looks like the following options are available:

  • a: mounting the contents cache folder as a tmpfs dir
  • b: setting Kolia cache limits to 0 at the repo level
  • c: using —no-caching when running kopia snapshot verify
  • d: is there another one ?
  1. What do you recommend to limit cache usage ?
2

Maybe it is not optimal but why to spend time on this? Just point cache to your local SSD and that’s it:)

1 Like
3

I prefer a software solution to a « hardware » one when available :slight_smile: and keep your solution in mind.

Perhaps I misunderstood / incorrect expectation about how kopia operates during « snapshot verify ». If so, it would be nice to know why « …/cache/contents »is used in the context of the original post.

PS: the size of the cache is not an issue (unless I intend to mount cache/contents to tmpfs), only its « constant » writes.

4

Maybe somebody more familiar with kopia inner workings can shed some light on this restore process aspect.

My approach is not to waste time on things which are not really show stoppers. Especially with projects like kopia - it is not maybe abandonware yet but any progress is very slow and authors/experts have abandoned answering forum questions years ago:)

So I would say use it as it is or seek some other solution.

1 Like
5

For now, I will try to Tmpfs mount the contents directory, set a hard limit on its size lower than the Tmpfs mount size (via kopia cache set).

6

I think Kopia is maturing and pace of development may slow down a bit imho.

Thank you @kapitainsky for taking the time to answer.

Will post results at some point in time.

1 Like
7

With Kopia cache content hard limit slightly smaller Tmpfs size mount of the same sir works well. Current hard cache limit is 5GB:

  • disk read speed is 30-100MB/s (skewed toward 30-40MB/s more often) compared to up to 20MB/s prior to the change.
  • disk writes in the kB/s range rather than MB/s.

Note: —file-paralelism 1 and —parralel 1 are used.

1 Like