Is there way to migrate to different crypto and splitter algorithms?

General
#1

I got the 0.8.0-beta1 and ran benchmark on it. The new Blake3 hashing is way better than Blake2 on my hardware. It was introduced in 0.7.1, after I’ve created most of my repositories. I would imagine Kopia will keep introducing new algorithms in the future, so is there a way to “upgrade” repositories to use them?

I understand crypto and splitter could be tied to repository, so to “change” them I’d have to create a new one. So how would I create a new repository with all the policies and snapshotted directories copied over from the old repository? It can be very tedious task to manually copy and paste each policy.

Could Kopia allow changing the algorithms if user deletes all snapshots but keep all the configurations?

Also, I’d imagine one day some algorithms will become outdated or having security risk to use so Kopia would obsolete and eventually remove them. What will be the migration path for that scenario?

#2

Can you try kopia snapshot migrate --all - this moves snapshots and policies to a new repo.

#3

Very nice! I added --overwrite-policies to overwrite the default global policy. With that it works perfectly! Thank you Jarek.

#4

Hi CrendKing, Would you be able to document the steps you took to achieve this? I am reading snapshot migrate | Kopia and unsure what the right structure would be and/or steps of operations.

In example, do I have to create the repo first? - I assume yes.
Do I have to put the same parameters, same passwords/encryption key? - I assume yes, but your post seems to indicate that you can use different parameters, i.e. encryption / splitter and the migrate feature will work.
How does the decrypt/re-encrypt work, is it downloading everything from the first repo, does it stuff, then uploads the new chunks to the new repo?
How does the --sources work? Does it asks for Path? Hostnames? can I select specific an individual snapshot other than the “latest” or do I have to migrate fully?

I’d like to see & understand your end to end scenario to understand how I can re-use pieces for mine.

Thanks!

#5

Snapshot migration works against destination repository (which must be connected and generally empty but that’s not strictly required).

You need to provide the configuration file for the old repository.

Typical usage is:

# connect to old repository 
$ kopia repo connect —config-file=/tmp/old.config …

# create new repository in another location 
$ kopia repo create …

# start the migration 
$ kopia snapshot migrate —all —source-config=/tmp/old.config

# now manually delete data from old repository location

This will use old repository as if it were a file system and perform regular snapshots of all data and migrate all snapshot manifests and policies. It uses configuration for the new repository to determine which hash/encryption/splitter to use.

#6

Awesome! Thanks for providing this content. It seems a little counter intuitive that kopia snapshot migrate only requires the “source” and not the destination… Especially if you have several connected repositories, it seemed unclear to me what destination would apply.

I will disconnect everything I have connected right now, and use your approach. Thanks again.

#7

Btw. You may want to tweak global policy on the destination before starting the migration. This lets you specify compression globally.

#8

Thanks Jarek - would you be able to document the --source flag? Does it work by hostname? by snapshot ID? manifest? Please clarify how this can be used. Ideally, I’d like to move things only when necessary, i.e. one host at a time.

#9

It’s username@hostname:/path

#10

Thanks. I clicked “Edit Page” on snapshot migrate | Kopia and it gave a 404 as it points to: https://github.com/kopia/kopia/edit/master/site/content/docs/Reference/Command-Line/Common/snapshot-migrate.md

If I can assist in putting your comments in the main doc, perhaps it may assist other users? Please let me know if I can assist.

#11

This part of documentation is in code: