Snapshot migrate v1 to v2

Support
1

I just migrated a repository to v2 and would like to share some unexpected results

:~# kopia snapshot migrate --policies --all --source-config=.config/kopia/kv1.config
 * 0 hashing, 0 hashed (0 B), 0 cached (0 B), uploaded 0 B, estimating...
Migration finished.
default (64KiB) - allocated 102(5) chunks freed 101 alive 1 max 5 free list high water mark: 4
:~$
:~$ rclone size /mnt/backup/kopia
Total objects: 5596
Total size: 116.102 GBytes (124664020487 Bytes)
:~$ rclone size /mnt/backup/kopia.v2/
Total objects: 11135
Total size: 227.851 GBytes (244652958596 Bytes)
  1. The size difference between repo1 and repo2, nearly double…
  2. What does the the error message in red at the end of the migration mean?

I did a kopia gc run and kopia gc run --full but no changes
Both the repositories have Compressor: “s2-parallel-8”.

2

Can you post kopia content stats from both repositories?

3

Repo v1

kopia content stats --config-file=.config/kopia/k1.conf
Count: 100431
Total Bytes: 123.6 GB
Average: 1.2 MB
Histogram:

        0 between 0 B and 10 B (total 0 B)
      226 between 10 B and 100 B (total 14.4 KB)
     1719 between 100 B and 1 KB (total 812.8 KB)
     1623 between 1 KB and 10 KB (total 7.6 MB)
     3739 between 10 KB and 100 KB (total 204.2 MB)
    48927 between 100 KB and 1 MB (total 27.5 GB)
    44197 between 1 MB and 10 MB (total 96 GB)
        0 between 10 MB and 100 MB (total 0 B)

Repo v2

 kopia content stats
Count: 100470
Total Bytes: 244.6 GB
Average: 2.4 MB
Histogram:

        0 between 0 B and 10 B (total 0 B)
      206 between 10 B and 100 B (total 12.9 KB)
     1226 between 100 B and 1 KB (total 585.6 KB)
     1037 between 1 KB and 10 KB (total 3.7 MB)
      782 between 10 KB and 100 KB (total 23.3 MB)
      240 between 100 KB and 1 MB (total 94.5 MB)
    96979 between 1 MB and 10 MB (total 244.5 GB)
        0 between 10 MB and 100 MB (total 0 B)
4

Did you have compression enabled on both repositories?

5

In v2 format you should see something similar to:

Count: 401633
Total Bytes: 697.4 GB
Total Packed: 568.9 GB (compression 18.4%)
By Method:
  (uncompressed)         count: 160283 size: 147.9 GB
  pgzip-best-speed       count: 241350 size: 549.5 GB packed: 421 GB compression: 23.4%
Average: 1.7 MB
Histogram:

        0 between 0 B and 10 B (total 0 B)
     6654 between 10 B and 100 B (total 472.7 KB)
   107324 between 100 B and 1 KB (total 47.4 MB)
    63772 between 1 KB and 10 KB (total 188.2 MB)
    14666 between 10 KB and 100 KB (total 559.2 MB)
    43786 between 100 KB and 1 MB (total 24 GB)
   165431 between 1 MB and 10 MB (total 544 GB)
        0 between 10 MB and 100 MB (total 0 B)
6

yes
Both the repositories have Compressor: “s2-parallel-8”.

7

yes it should have

I have another repository in windows with s2-parallel-8 compression and is see the output like yours in content stats.
But this repository v2 is on windows and it a fresh repositorykopia snap create ...... no migration.

The Repov2 outputs I have posted here was created this way

kopia policy set --global --compression=s2-parallel-8
for extensions in gns3project 7z rar zip bzip2 gz xz squashfs
do
   kopia policy set --global --add-never-compress=$extensions
done
kopia snapshot migrate --policies --all --source-config=.config/kopia/kv1.config

If i am not wrong it appears that there is no compression happening in snapshot migrate atleat for a destination repo which has no snapshots at all…

Wondering if i should recreate my v2 repository again check…

8

OK Done my research… This time on a Windows system with same results.

I just tested snapshot migrate with compression enabled on both repositories.
My guess was right…

Steps to Reproduce.

# disconnect
kopia repository disconnect
export KOPIA_PASSWORD="password for kopia repository"
# connect to old repository version 1
kopia repository connect filesystem --path /mnt/kv1 --config-file=k1.config
export KOPIA_PASSWORD="password for kopia repository2"
# create and conencto to repository version 2
kopia repository create filesystem --path /mnt/kv2
kopia repository connect filesystem --path /mnt/kv2
# setup policies for repository version 2
kopia policy set --global --compression=s2-parallel-8
for extensions in 7z rar zip bzip2 gz xz squashfs
do
    kopia policy set --global --add-never-compress=$extensions
done
# Start migration
kopia snapshot migrate --policies --all --source-config=$HOME/.config/kopia/k1.config
#check content status of repository2
kopia content stats

Can this be a bug or am I still missing something… :thinking:

Thanks :slight_smile:

9

@jkowalski

Is this a bug?

10

Yes, it was just fixed yesterday. Will be in v0.9.4 release, hopefully today.

11

You can test it using a build from

1 Like