Permission issues in .cache/kopia/

General Topics
1

Every now and then, I get this kind of error:

 Error: snapshot task: error flushing writer: error flushing contents: error flushing indexes: unable to add committed content: unable to open pack index "XXXX-XXXX-c1": unable to open file despite retries: open /var/lib/kopia/.cache/kopia/XXXXX/indexes/XXXX-XXXX-c1.sndx: permission denied

Due to reasons unknown, even though kopia runs on a separate user, sometimes there are rogue cache files that are owned by root (all the others are owned by kopia).

[11:27:33][root@kopia]:~# ps aux | grep [k]opia
kopia     429623  0.0 18.4 1917912 385936 ?      Ssl  paź29  12:18 /usr/bin/kopia server start --address=https://0.0.0.0:443 --tls-cert-file=/zbiornik/ssl/fullchain.pem --tls-key-file=/zbiornik/ssl/privkey.pem --ui --log-dir=/var/log/kopia --file-log-level=debug --log-level=info

[11:27:34][root@kopia]:~# id kopia
uid=999(kopia) gid=996(kopia) groups=996(kopia),100(users)
[11:32:06][root@kopia]:/var/lib/kopia# getent passwd kopia
kopia:x:999:996::/var/lib/kopia:/usr/sbin/nologin

[11:27:56][root@kopia]:~# fd --owner root . /var/lib/kopia/.cache/ | wc -l
4

Here’s how I’m hosting kopia:

[11:36:33][root@kopia]:/var/lib/kopia# systemctl status kopia-server.service
● kopia-server.service - Kopia Server (HTTPS on :443)
     Loaded: loaded (/etc/systemd/system/kopia-server.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-10-29 00:19:26 CET; 3 weeks 2 days ago
   Main PID: 429623 (kopia)
      Tasks: 15 (limit: 154116)
     Memory: 392.1M
        CPU: 12min 21.935s
     CGroup: /system.slice/kopia-server.service
             └─429623 /usr/bin/kopia server start --address=https://0.0.0.0:443 --tls-cert-file=/zbiornik/ssl/fullchain.pem --tls-key-file=/zbiornik/ssl/privkey.pem --ui --log-dir=/var/log/kopia --file-log-level=d>

lis 21 06:00:13 kopia kopia[726988]: [158B blob data]
lis 21 06:00:23 kopia kopia[727024]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
lis 21 06:00:23 kopia kopia[727024]:                                  Dload  Upload   Total   Spent    Left  Speed
lis 21 06:00:23 kopia kopia[727024]: [158B blob data]
lis 21 06:00:30 kopia kopia[429623]: ERROR snapshot error: snapshot task: error flushing writer: error flushing contents: error flushing indexes: unable to add committed content: unable to open pack index "xn0_9d7>
lis 21 06:01:42 kopia kopia[727046]:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
lis 21 06:01:42 kopia kopia[727046]:                                  Dload  Upload   Total   Spent    Left  Speed
lis 21 06:01:42 kopia kopia[727046]: [158B blob data]
lis 21 06:11:39 kopia kopia[429623]: WARN refresh error        {"err":"unable to refresh repository: error refreshing content index: unable to open pack index \"xxxx-xxxx>
lis 21 10:11:39 kopia kopia[429623]: WARN refresh error        {"err":"unable to refresh repository: error refreshing content index: unable to open pack index \"xxxx-xxxx>


[11:36:39][root@kopia]:/var/lib/kopia# cat /etc/systemd/system/kopia-server.service
[Unit]
Description=Kopia Server (HTTPS on :443)
After=network-online.target
Wants=network-online.target
# Don’t start if repo config is missing
ConditionPathExists=/var/lib/kopia/repository.config

[Service]
User=kopia
Group=kopia
EnvironmentFile=/etc/kopia/kopia-server.env

# Let the process bind to privileged port 443 without root
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

# Hardening
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
ReadWritePaths=/var/lib/kopia /var/log/kopia
# Read-only access to your cert directory
ReadOnlyPaths=/zbiornik/ssl

ExecStart=/usr/bin/kopia server start \
  --address=https://0.0.0.0:443 \
  --tls-cert-file=/zbiornik/ssl/fullchain.pem \
  --tls-key-file=/zbiornik/ssl/privkey.pem \
  --ui \
  --log-dir=/var/log/kopia \
  --file-log-level=debug \
  --log-level=info

# Graceful restarts
Restart=on-failure
RestartSec=2s
TimeoutStopSec=30s
# For many files
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
[11:36:41][root@kopia]:/var/lib/kopia#

Running chown kopia:kopia -R /var/lib/kopia/.cache/kopia solves the issue immediately…

Why is this happening?

2

Ok, I think I found the smoking gun.

I have a n8n workflow that reads email inbox that receives the Kopia notifications and enriches them before forwarding to my main email account.

The enrichment is basically running kopia diff CLI to list the actual files that have been added/removed.

And it runs that CLI as root - so my assumption is that this is what caused those rogue root-owner cache files.

The fix is to run kopia with sudo -u kopia.

(INB4 someone says that’s insecure to give n8n root access to kopia container - it’s done via fixed command in authorized_keys, so n8n can only run this kopia-diff script, nothing else)