NB.: all paths are trimmed to relative to repo dir. Also this may be a long first post, for a short answer. Can live with this 
Setup:
- r, l – clients (host and user names are the same for the given client)
- v – NAS, with separate SFTP accounts for r and l (the plan was SFTP-only, but on hold due to the problem).
- Both users belong to the kopia group.
- The repo directory was treated with (may have been redundant, was working out things mid-fight):
chgrp -R kopia . chmod -R g+ws . setfacl -m d:group:kopia:rwX . setfacl -m d:group:kopia . setfacl -m d:other::--- . - The plan was to separate the accesses to try insuring repo safety if the SSH key is compromised (which, let’s be honest, is a matter of time when using passwordless SSH keys). But a moot point, when both users have complete RW access to the whole repo. So probably this just breaks the whole scenario, but I still want to get to the bottom of this.
- I was sure that the repository can be used by multiple users (in kopia sense), confirmed here: Question: Kopia Repo vs Multiple Hosts
- The same about concurrent backups: Is it safe to backup concurrently to one S3 repo?
What happened:
- l was doing backups each 12h.
- Yesterday and the day before r had tried to do a initial snapshot, but was failing due to similar messages as now on l, but they were tried to be fixed with the treatment mentioned above.
- Today r was able to finish it’s snapshot and it can do another without problems.
- l was doing it’s snapshots without problems until today.
- Now each snapshot has the following errors
**Error:** unable to create policy getter: unable to get policies: unable to get parent policies: unable to find manifest for source $USER@l:/home/$USER: unable to load manifest contents: error loading manifest content: error getting cached content: unable to complete GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) despite 10 retries, last error: unrecognized error when opening SFTP file q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied.
Only two blobs are erroring:
/${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
/${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
13:09:24.190 uploading $USER@l:/home/$USER
13:09:24.190 reloading committed manifest contents: rev=220 last=0
13:09:24.190 listing manifest contents
13:09:24.195 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#0), sleeping for 100ms before retrying
13:09:24.195 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#0), sleeping for 100ms before retrying
13:09:24.298 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#1), sleeping for 150ms before retrying
13:09:24.298 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#1), sleeping for 150ms before retrying
13:09:24.451 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#2), sleeping for 225ms before retrying
13:09:24.452 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#2), sleeping for 225ms before retrying
13:09:24.679 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#3), sleeping for 337.5ms before retrying
13:09:24.680 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#3), sleeping for 337.5ms before retrying
13:09:25.020 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#4), sleeping for 506.25ms before retrying
13:09:25.020 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#4), sleeping for 506.25ms before retrying
13:09:25.529 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#5), sleeping for 759.375ms before retrying
13:09:25.529 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#5), sleeping for 759.375ms before retrying
13:09:26.295 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#6), sleeping for 1.1390625s before retrying
13:09:26.295 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#6), sleeping for 1.1390625s before retrying
13:09:27.438 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#7), sleeping for 1.70859375s before retrying
13:09:27.438 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#7), sleeping for 1.70859375s before retrying
13:09:29.151 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#8), sleeping for 2.562890625s before retrying
13:09:29.151 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#8), sleeping for 2.562890625s before retrying
13:09:31.718 got error unrecognized error when opening SFTP file /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f: permission denied when GetBlob(q513370e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f,0,-1) (#9), sleeping for 3.844335937s before retrying
13:09:31.718 got error unrecognized error when opening SFTP file /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f: permission denied when GetBlob(q101a850f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f,0,-1) (#9), sleeping for 3.844335937s before retrying
Now I can’t get down what could be the problem on the OS side. The files are owned by the other user, but the permissions are okay, there is no open FH’s.
v# ls -hal /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
ls -hal /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
-rw-rw----+ 1 r kopia 370K May 2 12:28 /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
-rw-rw----+ 1 r kopia 476K May 1 20:04 /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
v# getfacl /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
getfacl /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
getfacl: Removing leading '/' from absolute path names
# file: ${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
# owner: r
# group: kopia
user::rw-
group::rwx #effective:rw-
group:kopia:rwx #effective:rw-
mask::rw-
other::---
getfacl: Removing leading '/' from absolute path names
# file: ${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
# owner: r
# group: kopia
user::rw-
group::rwx #effective:rw-
group:kopia:rwx #effective:rw-
mask::rw-
other::---
v# fuser /${REPOPATH}/q10/1a8/50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f
fuser /${REPOPATH}/q51/337/0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f
v# lsof | egrep '50f3d475e7a9f37717d930ff108-s1f689fcca5717f3210f.f|0e9cd5c4d1dc1be22e4e6b572ab-s23376473e3c3640f10f.f'
v#
So was it was wrong to assume that the repo can be owned by multiple OS users with a common OS group?