Is there way to migrate to different crypto and splitter algorithms?

General
1

I got the 0.8.0-beta1 and ran benchmark on it. The new Blake3 hashing is way better than Blake2 on my hardware. It was introduced in 0.7.1, after I’ve created most of my repositories. I would imagine Kopia will keep introducing new algorithms in the future, so is there a way to “upgrade” repositories to use them?

I understand crypto and splitter could be tied to repository, so to “change” them I’d have to create a new one. So how would I create a new repository with all the policies and snapshotted directories copied over from the old repository? It can be very tedious task to manually copy and paste each policy.

Could Kopia allow changing the algorithms if user deletes all snapshots but keep all the configurations?

Also, I’d imagine one day some algorithms will become outdated or having security risk to use so Kopia would obsolete and eventually remove them. What will be the migration path for that scenario?

2

Can you try kopia snapshot migrate --all - this moves snapshots and policies to a new repo.

1 Like
3

Very nice! I added --overwrite-policies to overwrite the default global policy. With that it works perfectly! Thank you Jarek.

1 Like
4

Hi CrendKing, Would you be able to document the steps you took to achieve this? I am reading snapshot migrate | Kopia and unsure what the right structure would be and/or steps of operations.

In example, do I have to create the repo first? - I assume yes.
Do I have to put the same parameters, same passwords/encryption key? - I assume yes, but your post seems to indicate that you can use different parameters, i.e. encryption / splitter and the migrate feature will work.
How does the decrypt/re-encrypt work, is it downloading everything from the first repo, does it stuff, then uploads the new chunks to the new repo?
How does the --sources work? Does it asks for Path? Hostnames? can I select specific an individual snapshot other than the “latest” or do I have to migrate fully?

I’d like to see & understand your end to end scenario to understand how I can re-use pieces for mine.

Thanks!

5

Snapshot migration works against destination repository (which must be connected and generally empty but that’s not strictly required).

You need to provide the configuration file for the old repository.

Typical usage is:

# connect to old repository 
$ kopia repo connect —config-file=/tmp/old.config …

# create new repository in another location 
$ kopia repo create …

# start the migration 
$ kopia snapshot migrate —all —source-config=/tmp/old.config

# now manually delete data from old repository location

This will use old repository as if it were a file system and perform regular snapshots of all data and migrate all snapshot manifests and policies. It uses configuration for the new repository to determine which hash/encryption/splitter to use.

6

Awesome! Thanks for providing this content. It seems a little counter intuitive that kopia snapshot migrate only requires the “source” and not the destination… Especially if you have several connected repositories, it seemed unclear to me what destination would apply.

I will disconnect everything I have connected right now, and use your approach. Thanks again.

7

Btw. You may want to tweak global policy on the destination before starting the migration. This lets you specify compression globally.

8

Thanks Jarek - would you be able to document the --source flag? Does it work by hostname? by snapshot ID? manifest? Please clarify how this can be used. Ideally, I’d like to move things only when necessary, i.e. one host at a time.

9

It’s username@hostname:/path

10

Thanks. I clicked “Edit Page” on snapshot migrate | Kopia and it gave a 404 as it points to: https://github.com/kopia/kopia/edit/master/site/content/docs/Reference/Command-Line/Common/snapshot-migrate.md

If I can assist in putting your comments in the main doc, perhaps it may assist other users? Please let me know if I can assist.

11

This part of documentation is in code:

12

I’ve followed this instructions to try to migrate data encrypted using AES to a new repo using Chacha.

So I connected kopia to the AES-encrypted repo

[thib@ergaster ~]$ kopia repository connect s3 \
    --bucket=my-bucket \
    --access-key=REDACTED \
    --secret-access-key=redacted \
    --endpoint="s3.fr-par.scw.cloud" \
    --config-file=/home/thib/old.config
Enter password to open repository: 

Connected to repository.

NOTICE: Kopia will check for updates on GitHub every 7 days, starting 24 hours after first use.
To disable this behavior, set environment variable KOPIA_CHECK_FOR_UPDATES=false
Alternatively you can remove the file "/home/thib/old.config.update-info.json".

I then created a filesystem repo with the crypto parameters I needed

[thib@ergaster ~]$ kopia repo create filesystem \
    --block-hash=BLAKE2B-256-128 \
    --encryption=CHACHA20-POLY1305-HMAC-SHA256 \
    --path=/home/thib/kopia_chacha
Enter password to create new repository: 
Re-enter password for verification: 
Initializing repository with:
  block hash:          BLAKE2B-256-128
  encryption:          CHACHA20-POLY1305-HMAC-SHA256
  key derivation:      scrypt-65536-8-1
  splitter:            DYNAMIC-4M-BUZHASH
Connected to repository.

And then I attempted to migrate the data from my old repo to the new one

[thib@ergaster ~]$ kopia snapshot migrate --all \
    --source-config=/home/thib/old.config
    --parallel 16

When checking the status of my repo I can see that it is encrypted using AES

[thib@ergaster ~]$ kopia repo status
Config file:         /home/thib/.config/kopia/repository.config

Description:         Repository in Filesystem: /home/thib/kopia_chacha
Hostname:            ergaster
Username:            thib
Read-only:           false
Format blob cache:   15m0s

Storage type:        filesystem
Storage capacity:    1.3 TB
Storage available:   646.3 GB
Storage config:      {
                       "path": "/home/thib/kopia_chacha",
                       "fileMode": 384,
                       "dirMode": 448,
                       "dirShards": null
                     }

Unique ID:           eaa6041f654c5e926aa65442b5e80f6e8cf35c1db93573b596babf7cff8641d5
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
Format version:      3
Content compression: true
Password changes:    true
Max pack length:     21 MB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 0

Epoch refresh frequency: 20m0s
Epoch advance on:        20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin:    4h0m0s
Epoch checkpoint every:  7 epochs

Did I miss anything when performing the migration?