Sync only supports directly-connected repositories?

Support
1

I’m using Kopia in a docker container on my home server to back up various devices around the house to one repository.

I’d like to make a copy of this central repository at another location, and I’m currently testing that with my linux desktop. Obviously, I would prefer to pull from the server than push from it. Using rsync does not work because my server uses ZFS and the destination file system is EXT4.

I’ve tried connecting from the CLI. I’m able to connect as user@desktop, having created that user on the server already. However, trying to use sync-to results in the ERROR sync only supports directly-connected repositories. I’m guessing that’s because I’m not authenticated as the root server user (which was automatically assigned as abc@kopia. Any thoughts on how I can back up my server’s kopia repository?

2

To sync the repository from your home server to the linux desktop on the linux desktop by sync-to, a directly connection to both source and target repository of the home server is needed (that is, not by connecting via kopia server). Use other ways like sftp and webdav.

If you don’t want to expose the repository directly via sftp or webdav and want to protect the repository behind kopia server, you can use snapshot migrate command to copy snapshots from server to desktop.

Differences between sync-to and snapshot migrate:

  1. sync-to need directly connection, while snapshot migrate can copy snapshots from kopia server connection.
  2. sync-to can only mirror the whole state of whole repository, including creation and deletion of every snapshots (by every one), policy changes, snapshot changes, maintenance status, etc. Whether --delete is used, deletion of snapshots is synchronized too. snapshot migrate can copy any selection of snapshots or policies, and never delete snapshots that only exists on the target.
  3. sync-to copy blobs and indexes from the filesystem directly (that is why it can only mirror repository), while snapshot migrate read snapshots and policies from source repository and create same things in the target repository.
3

rsync does not care really about filesystem used. There is nothing stopping you from using it between ZFS and EXT4 filesystems.

I would only be not sure if using exFAT (as it does not support spare files). Still rsync would work but could not probably use delta updates.

4

Rsync doesn’t care, but I think Kopia does? When I copied the repository folder over directly and tried to open it, I got ERROR error connecting to repository: repository not initialized in the provided storage. I assumed this had to do with Jarek’s note on sharding.

5

No if you rsync local to local. You said ZFS to EXT4…

Sharding can only play part when you copy local ↔ some cloud (like S3). But then you can not use rsync for simple reason that it does not support cloud providers:)

6

Got it. Do you know what caused that error then?

7

This is weird… do you have kopia.repository.f and kopia.blobcfg.f files there? at destination.

I copied repos many times between different locations and never seen such error.

Are you trying to connect to it on the destination machine? or maybe on the source one? In that case maybe copia is confused that two repos have the same ID - the error might be side effect only.

8

Thanks for all this info! I want to mirror the repository, so it sounds like I want to use sync-to rather than snapshot migrate.

Just to confirm. You’re saying that a “direct connection” is NOT between kopia servers but rather from one kopia server to a target filesystem. In order to mirror my repository, I would need to initiate the sync from the repository source (in this case, my server). This would require access to the target filesystem via sftp, webdav etc. Is that all correct?

9

Yes. Need sftp or webdav. Rclone is OK too.

10

I tried again from scratch and did not get the error.

In this case, I accessed my server’s filesystem via SMB and copied over the repository folder using my file explorer application on my desktop. Then using the Kopia CLI on my desktop, I ran ``kopia repository connect filesystem`. This connected successfully, after which I was able to view and mount the snapshots normally.

Thanks for the tip! Very strange to get that error though. I wonder in which cases it’s supposed to show up.

11

I really think that this error in this case is just side affect of some other problem - hard to guess without exact details what/how you rsynced it

12

If rsync have mirror the repository correctly, kopia should be able to open it like the original repository. Could you check if here is any wrong with the commandline (like a wrong path, same path on different machine, rsync ignored some files, etc)

13

Closing the loop for anyone reading this in the future.

I believe that the ERROR sync only supports directly-connected repositories message was caused by an incomplete rsync operation that left the repository in a partially updated state. In my case, rsync failed because it did not have read permissions for every folder and file in the repository.

The official docker image doesn’t currently support PGID, PUID, or UMASK. When taking a snapshot, it’ll create files and folders that can’t be read by any user other than root. To ensure a successful transfer, you’ll either need to change the file permissions or give the remote access user sufficient privileges to copy the file.