Sync-to not detecting identical blobs after reversing direction

Support
1

Good morning,

I have a strange issue with repository synchronization: after sync-to from an external HDD to an SFTP location, then updating my repository location and taking new snapshots, my sync-to from the SFTP to HDD location is not recognizing any identical blobs.

I’d appreciate any help in understanding this - I can always clear the external HDD and re-sync, but it’s pretty slow and would take 20 hours, and I’m clearly missing something about how repository synchronization works. Perhaps a permissions issue?

Details follow:

> kopia --version
0.8.4 build: 940d10dfa29e771467b5d9d45412b889053c2ed1 from: kopia/kopia

With the current repository location as N:\kopia\BIGWHITE, I ran:

kopia repository sync-to from-config --file=kopia_BIGWHITE_SFTP.config --parallel=1 --delete

After a day of transfers, I changed the main repository to the SFTP location:

kopia repository connect from-config --file=kopia_BIGWHITE_SFTP.config

Then I deleted a few small-change snapshots, updated my local snapshots (a few hundred MB of changes) and some exclusion policies, and ran:

kopia maintenance run --full

Then I try to sync the changes back to my external drive, but none of the blobs are recognized (command aborted once copying started):

> kopia repository sync-to filesystem --path="N:\\kopia\\BIGWHITE" --parallel 1 --delete
Synchronizing repositories:
  Source:      SFTP nickersonm@alpine-btrfs
  Destination: Filesystem: N:\kopia\BIGWHITE
Looking for BLOBs to synchronize...
  Found 33518 BLOBs in the destination repository (767.2 GB)
  Found 33679 BLOBs (769.8 GB) in the source repository, 33679 (769.8 GB) to copy
  Found 0 BLOBs to delete (0 B), 0 in sync (0 B)
Copying...
  Copied 6 blobs (88.8 MB), Speed: 93.8 Mbit/s, ETA: 18h14m42s (2021-06-13 01:59:01 PDT)

I tried kopia repository repair filesystem --path="N:\\kopia\\BIGWHITE" but the format blob already exists.

Listing the snapshots shows many overlapping snapshots:

> kopia --config-file=kopia_BIGWHITE_extramobile.config snapshot list
Enter password to open repository:

nickersonm@bigwhite:C:\
  2021-06-10 18:09:50 PDT k1ac67439128a3fcb1e0ddf17867ad90b 157 GB drwxrwxrwx files:911826 dirs:139610 (latest-2,daily-2,hourly-2)
  2021-06-11 13:54:45 PDT k42e90a33a9503e80007293c027bb6034 156.7 GB drwxrwxrwx files:925405 dirs:141258 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

nickersonm@bigwhite:F:\
  2021-06-10 13:38:16 PDT kb3d8b15cae206c5fba7a3cb5f3068025 444.8 GB drwxrwxrwx files:367049 dirs:24482 (latest-3,hourly-3)
  2021-06-10 18:06:49 PDT k39bc91733b9afede770f2e2e6ccba4f7 444.8 GB drwxrwxrwx files:367049 dirs:24483 (latest-2,daily-2,hourly-2)
  2021-06-11 13:35:49 PDT keece7479ba0a60fed39ed21b4356c53d 499.9 GB drwxrwxrwx files:367762 dirs:24446 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

nickersonm@bigwhite:F:\My Documents
  2021-06-10 07:35:25 PDT k7d4807818005d39110ae6df5713a8a7f 112.6 GB drwxrwxrwx files:235717 dirs:14422 (latest-4,hourly-4)
  2021-06-10 13:35:30 PDT kdc54ab2d5d5aaee59a3273e26c939da9 112.6 GB drwxrwxrwx files:235721 dirs:14433 (latest-3,hourly-3)
  2021-06-10 18:03:59 PDT k89f91868862154f654cd1e3bf80ed9fc 112.6 GB drwxrwxrwx files:235721 dirs:14433 (latest-2,daily-2,hourly-2)
  2021-06-11 13:33:01 PDT k9a517b3149d69ea2153722ee646d61cf 112.5 GB drwxrwxrwx files:235675 dirs:14453 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

nickersonm@bigwhite:Z:\My Stuff
  2021-06-10 08:27:24 PDT k1eb846f426d1db1d1f7c9bf203a00a90 299 GB drwxrwxrwx files:99381 dirs:4738 (latest-4,hourly-4)
  + 1 identical snapshots until 2021-06-10 13:37:15 PDT
  2021-06-10 18:05:48 PDT ka1c11ff9193fac11d557d54d1bd67519 299 GB drwxrwxrwx files:99386 dirs:4739 (latest-2,daily-2,hourly-2)
  + 1 identical snapshots until 2021-06-11 13:34:40 PDT

(kopia_BIGWHITE_extramobile.config simply contains the "storage" block defining the "path": "N:\\kopia\\BIGWHITE" config).

> kopia snapshot list
nickersonm@bigwhite:C:\
  2021-06-10 18:09:50 PDT k1ac67439128a3fcb1e0ddf17867ad90b 157 GB drwxrwxrwx files:911826 dirs:139610 (latest-3,daily-3,hourly-3)
  2021-06-11 13:54:45 PDT k42e90a33a9503e80007293c027bb6034 156.7 GB drwxrwxrwx files:925405 dirs:141258 (latest-2,daily-2,hourly-2)
  2021-06-12 07:29:27 PDT k01b266b7859705a566fb5f73886929ef 157.3 GB drwxrwxrwx files:926595 dirs:141449 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

nickersonm@bigwhite:F:\
  2021-06-10 18:06:49 PDT k39bc91733b9afede770f2e2e6ccba4f7 444.8 GB drwxrwxrwx files:367049 dirs:24483 (latest-3,daily-3,hourly-3)
  2021-06-11 13:35:49 PDT keece7479ba0a60fed39ed21b4356c53d 499.9 GB drwxrwxrwx files:367762 dirs:24446 (latest-2,daily-2,hourly-2)
  2021-06-12 07:24:07 PDT kfb41c82d6abe2141424a85d6c8c626f1 499.9 GB drwxrwxrwx files:367761 dirs:24446 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

nickersonm@bigwhite:F:\My Documents
  2021-06-10 07:35:25 PDT k7d4807818005d39110ae6df5713a8a7f 112.6 GB drwxrwxrwx files:235717 dirs:14422 (latest-4,hourly-4)
  2021-06-10 18:03:59 PDT k89f91868862154f654cd1e3bf80ed9fc 112.6 GB drwxrwxrwx files:235721 dirs:14433 (latest-3,daily-3,hourly-3)
  2021-06-11 13:33:01 PDT k9a517b3149d69ea2153722ee646d61cf 112.5 GB drwxrwxrwx files:235675 dirs:14453 (latest-2,daily-2,hourly-2)
  2021-06-12 07:22:11 PDT kd5a79bf8b8b9bbe072d076c6feee4a75 102.1 GB drwxrwxrwx files:172495 dirs:14075 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

nickersonm@bigwhite:Z:\My Stuff
  2021-06-10 18:05:48 PDT ka1c11ff9193fac11d557d54d1bd67519 299 GB drwxrwxrwx files:99386 dirs:4739 (latest-3,daily-3,hourly-3)
  + 1 identical snapshots until 2021-06-11 13:34:40 PDT
  2021-06-12 07:23:17 PDT kb6f78476bbbda9e22174bb0fd6107d5f 299 GB drwxrwxrwx files:99387 dirs:4739 (latest-1,annual-1,monthly-1,weekly-1,daily-1,hourly-1)

I have also verified that the kopia.repository.f files in both locations are identical, and there are many identical blob files. Deleting the local cache did not change behavior.

Thanks for any assistance anyone may be able to provide!

- nickersonm

2

Brief update: there is no change when using samba instead of SFTP for the remote repository, so it’s not an SFTP issue.

3

Here’s a very minimal reproduction of this: I just have two external drives, one on the local PC, and one on a PC shared on my LAN. Both are Windows machines. I have synchronized one to the other; here I do a final sync-to and then reverse direction.

PS C:\Users\Michael Nickerson> kopia repository sync-to filesystem --path="\\10.10.10.1\MobileBackup\kopia\BIGWHITE" --parallel 1 --delete
Synchronizing repositories:
  Source:      Filesystem: N:/kopia/BIGWHITE
  Destination: Filesystem: \\10.10.10.1\MobileBackup\kopia\BIGWHITE
Looking for BLOBs to synchronize...
  Found 34511 BLOBs in the destination repository (788.8 GB)
  Found 34508 BLOBs (788.8 GB) in the source repository, 8 (81.8 KB) to copy
  Found 10 BLOBs to delete (39.4 MB), 34500 in sync (788.8 GB)
Copying...
  Copied 8 blobs (81.8 KB), Speed: -, ETA: unknown
PS C:\Users\Michael Nickerson> kopia repository connect filesystem --path="\\10.10.10.1\MobileBackup\kopia\BIGWHITE"
Enter password to open repository:

Connected to repository.

NOTICE: Kopia will check for updates on GitHub every 7 days, starting 24 hours after first use.
To disable this behavior, set environment variable KOPIA_CHECK_FOR_UPDATES=false
Alternatively you can remove the file "C:\Users\Michael Nickerson\AppData\Roaming\kopia\repository.config.update-info.json".
PS C:\Users\Michael Nickerson> kopia repository sync-to filesystem --path="N:\kopia\BIGWHITE" --parallel 1 --delete
Synchronizing repositories:
  Source:      Filesystem: \\10.10.10.1\MobileBackup\kopia\BIGWHITE
  Destination: Filesystem: N:\kopia\BIGWHITE
Looking for BLOBs to synchronize...
  Found 34508 BLOBs in the destination repository (788.8 GB)
  Found 16688 BLOBs (381.4 GB) in the source repository, 16688 (381.4 GB) to copy

(final command aborted before all blobs detected)

Let me know if there’s any useful debugging information I can provide. Thanks for any ideas anyone might have!

- nickersonm

4

With some further testing on a minimal reproducible example, it appears the reason is that --no-times is the default for sync-to instead of --times. At least for kopia running on Windows. Using rclone or --times solves my problem.

I opened a github issue for this.

- nickersonm