One repo, two SFTP accounts, repo is locked to a given client at a time

Support
1

NB.: all paths are trimmed to relative to repo dir. Also this may be a long first post, for a short answer. Can live with this :stuck_out_tongue:

Setup:

  • r, l – clients (host and user names are the same for the given client)
  • v – NAS, with separate SFTP accounts for r and l (the plan was SFTP-only, but on hold due to the problem).
    • Both users belong to the kopia group.
    • The repo directory was treated with (may have been redundant, was working out things mid-fight):
      chgrp -R kopia .
chmod -R g+ws .
setfacl -m d:group:kopia:rwX .
setfacl -m d:group:kopia .
setfacl -m d:other::--- .
    • The plan was to separate the accesses to try insuring repo safety if the SSH key is compromised (which, let’s be honest, is a matter of time when using passwordless SSH keys). But a moot point, when both users have complete RW access to the whole repo. So probably this just breaks the whole scenario, but I still want to get to the bottom of this.
  • I was sure that the repository can be used by multiple users (in kopia sense), confirmed here: Question: Kopia Repo vs Multiple Hosts
  • The same about concurrent backups: Is it safe to backup concurrently to one S3 repo?

What happened:

  • l was doing backups each 12h.
  • Yesterday and the day before r had tried to do a initial snapshot, but was failing due to similar messages as now on l, but they were tried to be fixed with the treatment mentioned above.
  • Today r was able to finish it’s snapshot and it can do another without problems.
  • l was doing it’s snapshots without problems until today.
  • Now each snapshot has the following errors

**Error:** unable to create policy getter: unable to get policies: unable to get parent policies: unable to find manifest for source $USER@l:/home/$USER: unable to load manifest contents: error loading manifest content: error getting cached content: unable to complete GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) despite 10 retries, last error: unrecognized error when opening SFTP file q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied.

Only two blobs are erroring:

/${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
/${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f

13:09:24.190 uploading $USER@l:/home/$USER
13:09:24.190 reloading committed manifest contents: rev=220 last=0
13:09:24.190 listing manifest contents
13:09:24.195 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#0), sleeping for 100ms before retrying
13:09:24.195 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#0), sleeping for 100ms before retrying
13:09:24.298 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#1), sleeping for 150ms before retrying
13:09:24.298 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#1), sleeping for 150ms before retrying
13:09:24.451 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#2), sleeping for 225ms before retrying
13:09:24.452 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#2), sleeping for 225ms before retrying
13:09:24.679 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#3), sleeping for 337.5ms before retrying
13:09:24.680 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#3), sleeping for 337.5ms before retrying
13:09:25.020 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#4), sleeping for 506.25ms before retrying
13:09:25.020 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#4), sleeping for 506.25ms before retrying
13:09:25.529 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#5), sleeping for 759.375ms before retrying
13:09:25.529 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#5), sleeping for 759.375ms before retrying
13:09:26.295 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#6), sleeping for 1.1390625s before retrying
13:09:26.295 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#6), sleeping for 1.1390625s before retrying
13:09:27.438 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#7), sleeping for 1.70859375s before retrying
13:09:27.438 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#7), sleeping for 1.70859375s before retrying
13:09:29.151 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#8), sleeping for 2.562890625s before retrying
13:09:29.151 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#8), sleeping for 2.562890625s before retrying
13:09:31.718 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#9), sleeping for 3.844335937s before retrying
13:09:31.718 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#9), sleeping for 3.844335937s before retrying

Now I can’t get down what could be the problem on the OS side. The files are owned by the other user, but the permissions are okay, there is no open FH’s.

v# ls -hal /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
ls -hal /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
-rw-rw----+ 1 r kopia 370K May  2 12:28 /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
-rw-rw----+ 1 r kopia 476K May  1 20:04 /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
v# getfacl /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
getfacl /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
getfacl: Removing leading '/' from absolute path names
# file: ${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
# owner: r
# group: kopia
user::rw-
group::rwx                    #effective:rw-
group:kopia:rwx         #effective:rw-
mask::rw-
other::---

getfacl: Removing leading '/' from absolute path names
# file: ${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
# owner: r
# group: kopia
user::rw-
group::rwx                    #effective:rw-
group:kopia:rwx         #effective:rw-
mask::rw-
other::---

v# fuser /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
fuser /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
v# lsof | egrep '50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f|0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f'
v#

So was it was wrong to assume that the repo can be owned by multiple OS users with a common OS group?

2

IDK if I should boost it or something, as the post just got three days penalty bench for something.

3

It’s advantage of kopia to backup to the same repository from multiple OS/users. The only thing you doing IMHO wrong is to giving the same credentials to both clients, that are in fact are different sources. kopia supports concept of ACL(access control list) and users. You should create separate accounts in repository for each user, but internally you would still benefit from deduplication for all clients. Also, kopia supports ACL that allows to set repository in APPEND mode, that will safe backup in case user’s computer get infected and virus or malicious person might overwrite/delete backups.

4

The only thing you doing IMHO wrong is to giving the same credentials to both clients, that are in fact are different sources.

I’m not providing the same credentials, except the repository password. They had separate SSH keys for user separation. Which, again, is moot when being forced to use passwordless keys or not supporting ssh-agent (maybe with external ssh, but I’m moving to kopia server), to the same FS with RW permissions, something my inner secops voice can’t stop screaming about.

Isn’t APPEND ACL mode a feature of the kopia server anyway?

Anyway it seems the only way to have multiple users on a single repo is to use a kopia server.

5

Additional 2 fun things: after 2 days of errors kopia was able to make two snapshots two days ago somehow.

Unfortunately I can’t list the details, because my NAS went FUBAR yesterday and I’m working on recovering.

The second “fun” thing, that all the time the NAS was gone from the network kopia is trying to connect to it. Racked in 90k seconds. Yet I can’t upload a screenshot here.

6

You have to have separate kopia’s user accounts, not a ssh account.

Yes, that’s what I’m talking about - users of kopia server