SOLVED 1) Backing up the cachingDirectory beneficial? 2) cacheDirectory inner working?

[bbccdd]

1. Which parts of the cacheDirectory might be most beneficial to keep?
   “It is always safe to remove files from cache.” That is according to the caching documentation.

Yet, which types of caching could be beneficial to keep?

Context: I’m using rclone for a nightly clone of the full repo, but synchronizing caching data to pay-per-use remote storage does not make sense, or does it?

2. Why is a new directory created inside the cacheDirectory with a name such as ‘af3897f0a7c79755’ while the cacheDirectory already contains cashing information (/rust/kopia/1/cache/) based on this config file:

{
  "storage": {
    "type": "filesystem",
    "config": {
      "path": "/rust/kopia/1",
      "fileMode": 384,
      "dirMode": 448,
      "dirShards": null
    }
  },
  "caching": {
    "cacheDirectory": "/rust/kopia/1/cache",
    "maxCacheSize": 5242880000,
    "maxMetadataCacheSize": 5242880000,
    "maxListCacheDuration": 30
  },
  "hostname": "user",
  "username": "secret",
  "description": "Repository in Filesystem: /rust/kopia/1",
  "enableActions": false,
  "formatBlobCacheDuration": 900000000000
}

user@host:~# ls /rust/kopia/1/cache/
CACHEDIR.TAG  af3897f0a7c79755  blob-list  cli-logs  content-logs  contents  index-blobs  indexes  kopia.repository  metadata  own-writes
user@host:~# ls /rust/kopia/1/cache/af3897f0a7c79755/
CACHEDIR.TAG  blob-list  contents  index-blobs  indexes  kopia.repository  metadata  own-writes

[basldfalksjdf]

@bbccdd

Context: I’m using rclone for a nightly clone of the full repo, but synchronizing caching data to pay-per-use remote storage does not make sense, or does it?

Not sure I understand.

So, you have Repo A, where you use Kopia to store all your snapshots. Then you want to use Rclone to copy Repo A to Cloud B. Is that correct? If so, I don’t think you should worry about the cache. Kopia does not store the cache in its repositories. Cache is only stored on the local machine that runs Kopia. If you are copying Repo A using Rclone and not copying your local machine, then Rclone won’t copy the cache.

…But looking at your config file, it looks like you are storing the cache in the same directory as the repo. Not sure why. Nonetheless, it is safe to tell Rclone to completely ignore /rust/kopia/1/cache and copy the remainder of your repo /rust/kopia/1. You don’t need to copy any of the cache.

[bbccdd]

@basldfalksjdf

Nonetheless, it is safe to tell Rclone to completely ignore /rust/kopia/1/cache and copy the remainder of your repo /rust/kopia/1. You don’t need to copy any of the cache.

Hi @basldfalksjdf thank you for taking the time to respond. You’ve resolved the issue with that statement, I’ll exclude the cachingDirectory from rclone.

[basldfalksjdf]

Glad to hear it!

By the way, are you using Rclone to copy your repo to some place that Kopia does not support? If you are using Rclone to copy the repo to a cloud storage that Kopia does support, you don’t need to use Rclone. Just use Kopia’s sync-to command – it literally copies the contents of a repo to another repo.

[bbccdd]

The reason for choosing rclone was for a) performance and b) to support OneDrive. But that was about 1,5 years ago. So, when moving to full ransomware protection I’ll also test with the performance of build-in sync-to commands (at least for backblaze which is the primary destination).

a) Performance

@jkowalski

Kopia does not (yet) utilize any parallelism during restore (unlike upload which is heavily parallelized) . This means that with a high-latency store (like OneDrive) you may see long restore times, in particular when you have tons of small files.

b) To support OneDrive as a destination.
To also sync the kopia repository server that serves multiple clients to MS OneDrive (where I have 1TB storage that does not require paying extra for) which is not supported by kopia itself.

[basldfalksjdf]

A few things:

• Using your current workflow, you will NOT be able to restore directly from OneDrive if you setup Kopia to store snapshots in a local filesystem repo and then use Rclone to copy the files to OneDrive. If you ever need to restore from the files in OneDrive, you will need to copy them back to the local filesystem before Kopia will be able to restore them.

• If you want to be able to restore directly from OneDrive, you can setup a Kopia repository on OneDrive through Kopia’s Rclone support. Then you can use sync-to to copy Repo A to the Rclone/OneDrive repo. In this situation, you will be able to use Kopia to restore directly from the Rclone/OneDrive repo.

• I believe parallelism has been added to Kopia restore now, although note that Rclone-based repos will still be slower than non-Rclone repos.

[bbccdd]

@basldfalksjdf

• Using your current workflow, you will NOT be able to restore directly from OneDrive if you setup Kopia to store snapshots in a local filesystem repo and then use Rclone to copy the files to OneDrive. If you ever need to restore from the files in OneDrive, you will need to copy them back to the local filesystem before Kopia will be able to restore them.

I know. Sub-optimal, but still an extra failsafe.

@basldfalksjdf

If you want to be able to restore directly from OneDrive, you can setup a Kopia repository on OneDrive through Kopia’s Rclone support. Then you can use sync-to to copy Repo A to the Rclone/OneDrive repo. In this situation, you will be able to use Kopia to restore directly from the Rclone/OneDrive repo.

Cool! I’ll look into that, thanks!