I am currently synchronizig a Kopia server repo to a Wasabi S3 bucket. The sync is probably halfway through and kopia.repo/kopia.maintenance have already been synched. However, looking at the bucket, the two files don’t have the suffix .f which might be deliberately the case to prevent anyone from messing with the repo until the synch has finished.
If that’s the case, then it would be no wonder, that I cannot connect to the repo and I am getting a password refused error.
Well, the S3 (that includes Wasabi), GCS, B2 and Azure backends don’t use .f suffix nor directory structures because the underlying stores provide atomic writes and can scale horizontally. ‘.f’ is only needed to distinguish partially-written files from fully written ones (have .f) and directory structure helps reduce local filesystem load (also not needed on cloud providers)
To sync repositories between filesystem and cloud you need to use kopia repo sync-to and not regular file sync tool (well you can, but you must flatten the structure and remove ‘.f’ extension from the destination). Sync-to provides this automatically - it has atomic writes and incremental synchronization.
Yes, I know… and I am actually running kopia repo-sync-to. I was under the impression, that I would get a fully operable repo, to which I could connect my KopiaUI and e.g. restore files from, when I am not on my local LAN.
Hi @jkowalski , can you please either confirm or deny, that it’s possible to actually login to a cloned repo? My sync-to S3 job has finished (after 6 days), but I cannot login to that repo, since Kopia refuses to accept the repo password from the source repo and I thought that this should work.
There is actually a test for it so it is definitely working.
Can you post your commands and log snippets that shows the failure?
Sure - I first tried that in Kopia UI, but since that wasn’t working, I tried with kopia on CentOS as well…
[root@kopia repos]# kopia repo connect s3 --secret-access-key=<super secret key> --access-key=<access key> --endpoint=s3.eu-central-1.wasabisys.com --bucket=kopiast7a
Enter password to open repository:
failed to open repository: invalid repository password
I tried to enter the original repo password, which I used to initially create the repo on disk… As you can see, there’s not much to it…
Can you add --log-level=debug and see if there’s anything there?
Actually can you list files in the bucket? (I want to make sure kopia.repository is there as are N, P and Q blobs)
Not much…
Enter password to open repository:
Creating cache directory ‘…/…/.cache/kopia/ea27647e8f1ea2f2’ with max size 5242880000
failed to open repository: invalid repository password
kopia: error: error connecting to repository: invalid repository password, try --help
I can get you a screenshot, since I don’t have the FS mounted anywhere, but I can confirm that the files kopia.maintenance, kopie.repository and a whopping 22k of blobs are in the bucket.
What about s3 bucket listing?
Let me quickly install s3fs on my kopia server…
Here it comes…
insgesamt 489371797
-rw-r-----. 1 root root 1818 19. Feb 19:41 kopia.maintenance
-rw-r-----. 1 root root 686 19. Feb 19:04 kopia.repository
-rw-r-----. 1 root root 172 19. Feb 19:41 ld42817ea6e69dedf4006d788cdd46b7d
-rw-r-----. 1 root root 1051 19. Feb 19:41 m57bbfadd1db3c59ab1dcc2cceadbf574
-rw-r-----. 1 root root 925 19. Feb 19:41 m8e5e59ecb0bf78688f3d938478d589bc
-rw-r-----. 1 root root 23943524 19. Feb 19:42 n285b66cd64cd1a608b107938a9e21fc4
-rw-r-----. 1 root root 1050549 19. Feb 19:42 n2bf868fd09a6020f46ad7888482ef52e
-rw-r-----. 1 root root 101206 19. Feb 19:42 n3120f87459fc6df10acc13ffc057869a-s40e5a91d37da0236101
-rw-r-----. 1 root root 159 19. Feb 19:42 n35be7ea268659487d050fe3cba290705-s0eaa24ede5835fca101
-rw-r-----. 1 root root 159 19. Feb 19:42 n398fb8922e5f871f096910746cdf3c34-sc4b9fbeff9eb2fd1101
-rw-r-----. 1 root root 159 19. Feb 19:42 n3a19c9666beb894e9967aa67cf699d99-s3a3aaee9fe97c8ab101
-rw-r-----. 1 root root 568832 19. Feb 19:42 n487a7b7c921d828128af354105aefcb9
-rw-r-----. 1 root root 1232 19. Feb 19:42 n4cd038485a2e3b40a5a61c3248545a5f-s15464e890d1dec2c101
-rw-r-----. 1 root root 1158 19. Feb 19:42 n510584075ab347d3ba1f4c00a1d09c9f-s672fe708613c1f45101
-rw-r-----. 1 root root 1116820 19. Feb 19:42 n52804080587b1460b4215d6b7f761717
-rw-r-----. 1 root root 159 19. Feb 19:42 n6d602eba469dab99fd9ed018dbab2949-s1dcc464f8eafc648101
-rw-r-----. 1 root root 25460478 19. Feb 19:42 n6f9359b1cf0a47c08b18b05921df185d
-rw-r-----. 1 root root 1158 19. Feb 19:42 n8007b0b258305329b4066fdc8375623e-s91dce93aead4e5fb101
-rw-r-----. 1 root root 568812 19. Feb 19:42 n88ca05a3e41d1ccf86d3cdf9d04ed3c4
-rw-r-----. 1 root root 1306 19. Feb 19:42 na359ff00a6e7d402c099e7f37477492f-s86dea195fc1af5e6101
-rw-r-----. 1 root root 1158 19. Feb 19:42 nb8f0dae5ae67f19aa3b9837276cc8c7a-s6fcd55d087e7b1ef101
-rw-r-----. 1 root root 21524943 19. Feb 19:43 p0003307cf7ca82dee93660e22b191225
-rw-r-----. 1 root root 22979578 19. Feb 19:43 p00034927f8cc5a75ee9ab875cde94e63
-rw-r-----. 1 root root 22700421 19. Feb 19:43 p0003890735ae38bdf8f6667a93b4d743
-rw-r-----. 1 root root 27808059 19. Feb 19:44 p000442bc7626b9d39cb749aa3f61bbb3
-rw-r-----. 1 root root 20992574 19. Feb 19:44 p000484db5a659df6bd6a0bdd191016be
-rw-r-----. 1 root root 21303685 19. Feb 19:44 p000951a353ad4676bdee0498989359b7
-rw-r-----. 1 root root 22049712 19. Feb 19:45 p000cf581dc85530f168725fe9e73f33d
-rw-r-----. 1 root root 22585955 19. Feb 19:45 p000d53d2b62210a4437a288a49f9b8cd
-rw-r-----. 1 root root 21087078 19. Feb 19:46 p000e93eb4ff2f59b477a8c89fbfa8ffb
-rw-r-----. 1 root root 23704015 19. Feb 19:46 p00132ed7cf396abdfd8d4d47ea773c20
-rw-r-----. 1 root root 21414869 19. Feb 19:46 p00155b8c46d165d8c8dfbaa62af7ae05
-rw-r-----. 1 root root 25993531 19. Feb 19:47 p001eacc4d5eee59b51f7a411b78d6841
-rw-r-----. 1 root root 20996411 19. Feb 19:47 p001fef55122c70d5e8c57e5ea0d9d5a3
-rw-r-----. 1 root root 24592773 19. Feb 19:47 p0021051b9086126d6e9ae2e91eb823e2
-rw-r-----. 1 root root 28828496 19. Feb 19:48 p0022f9b6c24120a945863a598391e710
-rw-r-----. 1 root root 22095175 19. Feb 19:48 p002327924d168e10f6049b038a9d22f9
…
-rw-r-----. 1 root root 21213127 25. Feb 11:34 qb1f5eea510c0ce01573d392d5393c94b
-rw-r-----. 1 root root 20506976 25. Feb 11:34 qbd4443ba1357336eacfffe84a9009575
-rw-r-----. 1 root root 13513762 25. Feb 11:34 qbdb9284b1e6623f4de5b8f0b262b5a7b
-rw-r-----. 1 root root 21908879 25. Feb 11:35 qc2cf51be643b70fed2b855eb76189037
-rw-r-----. 1 root root 21332235 25. Feb 11:35 qd3628756d728335bb5aed21726423bc9
-rw-r-----. 1 root root 16771956 25. Feb 11:35 qd6f3d7c99221b97de1e69c29b28f7da1-s40e5a91d37da0236101
-rw-r-----. 1 root root 21400456 25. Feb 11:35 qf2eefd09552dd43bb4edf8bcd7691e6f
-rw-r-----. 1 root root 21307900 25. Feb 11:36 qfa014928dc72a8928c426f57fe074ba2
-rw-r-----. 1 root root 21380292 25. Feb 11:36 qfa32b8c32865d08b5161ba07dfc9235e
-rw-r-----. 1 root root 197 25. Feb 11:36 s8df08890a4a997d5812d015b00a68a6a-sfad35155300abc77101
I checked the kopia.repository and kopia.repository.f files in my S3 bucket and in my local repo and they are indeed identical…
[root@kopia s3]# sha256sum kopia.repository
104cfca10d961d403d12c93f15824416f1fa587cb2f7b85879abdd884c5e2775 kopia.repository
[root@kopia s3]# sha256sum kopia.repository.bak
104cfca10d961d403d12c93f15824416f1fa587cb2f7b85879abdd884c5e2775 kopia.repository.bak
However, kopia doesn’t accept the password it accepts for the local repo…
The only explanation I have is that Kopia is trying to connect somewhere else. Did you specify the right endpoint and region?
If that would be the case, then sync-to shouldn’t work either, should it?
And this shouldn’t work as well…
[root@kopia repos]# kopia repository sync-to s3 --dry-run --max-upload-speed=1048576 --secret-access-key=<secret key> --access-key=<access key> --endpoint="s3.eu-central-1.wasabisys.com" --bucket=kopiast7a
Synchronizing repositories:
Source: Filesystem: /mnt/kopia/ST7A
Destination: S3: s3.eu-central-1.wasabisys.com kopiast7a
NOTE: By default no BLOBs are deleted, pass --delete to allow it.
Looking for BLOBs to synchronize...
Found 21701 BLOBs in the destination repository (501.1 GB)
Found 24721 BLOBs (570.8 GB) in the source repository, 3040 (69.7 GB) to copy
Found 0 BLOBs to delete (0 B), 21681 in sync (501.1 GB)
I also ran tcpdump when trying to connect and it showed traffic to the appropriate hosts…
Can you create repository in another bucket on the same endpoint?
Sure… will get back when I tried that.
btw, can you hop on slack? may be faster this way
Looks like the connection handler to S3 needs some kind of escapeing for the repo password. I created an issue in GH:
Can’t connect to S3 repo, when repo password contains “$” as character
This turned out to be shell escaping password passed via cmdline.
Note to future readers: avoid passing password vis command line or at least use single quotes around password so that shell won’t mess with it.