Containerized Kopia server setup

I spent some time over the last couple of days researching how to get Kopia up and running for my setup. I’m sharing here for feedback/advice and in hopes that it may help someone if they want a similar configuration.

I used Repository server via Docker - #2 by jkowalski as the starting spot for this configuration.

Configuration description

  1. Kopia running on my local Ubuntu-based NAS (called metal-mind), taking snapshots on a regular basis of all files on the NAS.
  2. Kopia running in a container so its interaction with the host is obvious and self-documented.
  3. Kopia UI exposed to my LAN so I can have a convenient way of checking up on it if I’m curious.
  4. Backup destination (repository) is BackBlaze’s B2.

Directory Setup

mkdir /home/ubuntu/kopia
cd /home/ubuntu/kopia
mkdir {cache,config,logs}
chown 65532:65532 {cache,config,logs} # kopia container runs in rootless mode

/home/ubuntu/kopia/docker-compose.yml

version: '3.7'
services:
    kopia:
        image: kopia/kopia:latest
        hostname: metal-mind
        restart: unless-stopped
        ports:
            - 51515:51515
        environment:
            KOPIA_PASSWORD: SuperSecretRepositoryPassword
            TZ: America/Los_Angeles
        volumes:
            - /home/ubuntu/kopia/config:/app/config
            - /home/ubuntu/kopia/cache:/app/cache
            - /home/ubuntu/kopia/cache:/app/logs
            - /media/backup:/app/backup:ro
        entrypoint: ["/app/kopia", "server", "--insecure", "--address=0.0.0.0:51515", "--override-username=kopia@metal-mind", "--server-username=kopia@metal-mind", "--server-password=SuperSecretPasswordForTheWebUI"]

Now you can docker-compose up -d and Kopia server will run. The WebUI should be accessible via http://metal-mind:51515, and you can login with the username and password from the bottom of the above docker-compose.yml. From here you can configure your repository and snapshots through the UI.

If you need to use Kopia on the CLI, then you need to get the ID of the running container so you can issue commands within it.

Get Kopia’s Container ID

ubuntu@container-host:~/kopia$ docker ps
CONTAINER ID   IMAGE                               COMMAND                  CREATED        STATUS                PORTS                                                                                                                                                                      NAMES
d2f3af390431   kopia/kopia:latest                  "/app/kopia server -…"   13 hours ago   Up 13 hours           0.0.0.0:51515->51515/tcp                                                                                                                                                   kopia_kopia_1
7f9fced43cf0   nginx:latest                        "/docker-entrypoint.…"   13 hours ago   Up 13 hours           0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                                                                   nginx
...

Issue Kopia Commands
Kopia appears to come bundled with basically nothing in its image, including a shell. That means we’ll have to use docker exec for each individual command we want to issue to kopia instead of just launching an interactive shell inside the container. This stripped down container also means we don’t have access to ls or any other standard tools to examine or debug the system from the container’s point of view.

docker exec -t d2f3af390431 /app/kopia --help
docker exec -t d2f3af390431 /app/kopia policy set --global --compression=zstd

Restoration
You can now install Kopia on any other computer and connect to the same repository (Backblaze’s B2 in my case) and see any snapshots created by the server. Make sure to set the filter dropdown in the upper left corner to “All Snapshots”. While this is a good way of restoring files, I’m planning on doing any administration via the server UI (http://metal-mind:51515).

Nginx
Finally, I configured Nginx to reverse proxy to Kopia. Since there are lots of guides for getting Nginx up and running, I’ll just post my config:

upstream kopia_backend {
    server metal-mind:51515;
    keepalive 32;
}

server {
    include ssl.conf;
    server_name kopia.my-domain.com;

    # Don't expose Kopia to anything other than my local network
    allow 192.168.100.0/24;
    deny all;

    #Forward real IP and host
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    location / {
        proxy_pass http://kopia_backend;
    }
}
8 Likes

Nice! I am going to piggyback here to show the equivalent configuration for an Unraid container:

First, create httpaswd credentials file as described in documentation:

htpasswd -c htpassword <your user name/email address>

“Add container” with the following settings:

  • Open port: 51515
  • Mount backup root: path → /mnt/user:/backuproot
  • Mount config: path → /mnt/user/appdata/Kopia/config:/app/config
  • Mount cache: path → /mnt/user/appdata/Kopia/cache:/app/cache
  • Password: Environment variable → KOPIA_PASSWORD=xxxx
  • Post Arguments: server --insecure --htpasswd-file /app/htpasswd --address 0.0.0.0:51515 -server-username=<your user name/email address>
2 Likes

BTW, I’ll be happy to include those on kopia.io website. I’ll be happy to review patches that modify site/ directory to add this content and/or links to this forum post.

Hi @cyansmoker I am not sure I am following the steps to add to Unraid - would you be able to elaborate on your setup?.. i’ve been trying to have a local kopia running on Unraid and resorted to use the CLI and supplying password manually, but sounds like you have a container running? I would love to know how this can work. Thanks!

I simply went to Docker > ADD CONTAINER then most of the information in my post can be added by clicking + Add another Path, Port, Variable, Label or Device selecting the correct ‘Config Type’ and entering the information. I am not sure what else makes this difficult?

For some reason I’m having nothing but permissions issues trying to run the server in a Docker container under OMV 5 (Debian). For example - here are the permissions on the empty logs folder before starting the container for the first time:

drwxr-xr-x 2 65532 65532 4096 Jul 16 22:04 logs

As soon as I start the container it seems to start fine with no errors and it creates a “cli-log” folder under logs with the following permissions:

drwx------ 2 65532 65532 4096 Jul 16 22:07 cli-logs

So at server start it seems to have no issues accessing/creating files and folders but when I try and create a repo it can’t create any files (under any folder, logs or data):

Unable to create logs directory: mkdir /app/logs/cli-logs: permission denied
Unable to create logs directory: mkdir /app/logs/content-logs: permission denied
unable to open log file: open /app/logs/cli-logs/kopia-20210717-030918-1-repository-create-filesystem.log: no such file or directory
unable to read log directory: open /app/logs/cli-logs: no such file or directory
unable to read log directory: open /app/logs/content-logs: no such file or directory

And then it bombs out trying to create the necessary files in the data folder:

kopia: error: cannot initialize repository: unable to write format blob: unable to write format blob: unable to complete PutBlobInPath:/app/data/kopia.repository.f despite 10 retries, try --help

Am I doing something dumb and missing a step here? I got this running last weekend under TrueNAS Scale and didn’t have this issue.

EDIT:

Ignore this. I was doing something dumb. When I did this under TrueNAS Scale last weekend I created some shell scripts to invoke the commands on the server (using docker run). Well I forgot to change the volume mounts in that shell script to the new locations on the OMV box so when trying to create the repo it was mounting to non-existing folder locations. Ugh.

Do I put htpasswd file first in its folder?
Please see pic attached, if you see some wrongs to guide me, docker doesn’t start!

With this post arguments :
server --insecure --htpasswd-file /app/htpasswd --address 0.0.0.0:51515 -server-username=<your user name/email address>

I got error:
kopia: error: unknown short flag ‘-s’, try --help

you need to use long flag name with two dashes (--server-username not -server-username)

1 Like

Thank you very much, Now the correct post arguments like this:

server --insecure --htpasswd-file /app/htpasswd/.htpasswd --address 0.0.0.0:51515 --server-username=xxxx

I had to correct htpasswd path .

Now the docker start but without web-ui
with this log:

OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "sh": executable file not found in $PATH: unknown

Update: I successfully run the webui by change network type from bridge to br0 (it is like now under router ip not under unraid ip)

How I put my Google drive token in it?

Hello,
I have pretty much the exact same configuration running Kopia server in an LXC container (run by user kopia, on port 51515) proxied by HAProxy (which takes care of TLS for domain say kopia.domain.tld)
I can access the server Kopia UI from the browser (https://kopia.domain.tld) and manage the repository. However, I cannot access the server repository from command line running:

kopia repository connect server --url=https://kopia.domain.tld:443 --server-cert-fingerprint=<fingerprint> --password=<password>

This outputs the following error:

021-10-15 18:40:47.319294 I [logger.go:244] Connecting to server 'https://kopia.domain.tld:443' as '<$USER>@<hostname>...
2021-10-15 18:40:47.319494 D [logger.go:254] Creating cache directory '/home/<$USER>/.cache/kopia/1b0e0fd7e3249467' with max size 5242880000
2021-10-15 18:40:47.563462 D [logger.go:254] establishing new GRPC streaming session (purpose=)
2021-10-15 18:40:47.621536 D [logger.go:254] GRPC stream read loop terminated with rpc error: code = Unauthenticated desc = unexpected HTTP status code received from server: 401 (Unauthorized); transport: received unexpected content-type "text/plain; charset=utf-8"
2021-10-15 18:40:47.621863 E [logger.go:214] failed to open repository: unable to establish session for purpose=: error establishing session: unable to initialize session: rpc error: code = Unauthenticated desc = unexpected HTTP status code received from server: 401 (Unauthorized); transport: received unexpected content-type "text/plain; charset=utf-8": EOF

I have tried [to] --override-username and --override-hostname using other credentials created beforehand on the Kopia server using kopia server users add <username>, in vain.

Besides, the proxy log outputs the following 401 error:

Oct 15 20:29:39 hostname haproxy[25806]: ::ffff:<remote ip>:40326 [15/Oct/2021:20:29:39.410] main~ KOPIA/KOPIA 0/0/0/1/7 401 221 - - CD-- 2/2/0/0/0 0/0 "POST https://kopia.domain.tld:443/kopia_repository.KopiaRepository/Session HTTP/2.0"

Any hints?
Thanks a lot for your help.

You need to proxy grpc in addition to http. That usually helps.

Well I have set HAProxy to use HTTP2.0 procotol which embeds gRPC compatibility, but it won’t work.
Maybe something is wrong on the HAProxy side (see grpc client deadline exceeded and cancellations do not work while proxying through haproxy as reset frame is not forwarded to backend · Issue #172 · haproxy/haproxy · GitHub for instance). Actually and sadly, it goes a bit beyond my understanding and knowledge of proxying and server configuration.
What strikes me is that everything works through the browser but won’t work using server repository connection command line.
The HAProxy error that prints shows ”CD” keyword which from the (old) documentation (I cannot find the equivalent for HAProxy 2.2 version I use) corresponds to:

      CD  The client unexpectedly aborted during data transfer. This is either
          caused by a browser crash, or by a keep-alive session between the
          server and the client terminated first by the client.

Maybe this can help:

❯ curl -v https://<kopia.domain.tld>:443
*   Trying <ip-address>:443...
* Connected to <kopia.domain.tld> (<ip-address>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=<kopia.domain.tld>
*  start date: Oct 12 09:06:48 2021 GMT
*  expire date: Jan 10 09:06:47 2022 GMT
*  subjectAltName: host "<kopia.domain.tld>" matched cert's "<kopia.domain.tld>"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56323e8e6560)
> GET / HTTP/2
> Host: <kopia.domain.tld>
> user-agent: curl/7.74.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 401
< content-type: text/plain; charset=utf-8
< www-authenticate: Basic realm="Kopia"
< x-content-type-options: nosniff
< date: Sat, 16 Oct 2021 10:04:33 GMT
< content-length: 22
<
Missing credentials.

* Connection #0 to host <kopia.domain.tld> left intact

While using htpasswd connection credentials:

❯ curl -v https://<kopia.domain.tld>:443 -u "kopia:KopiaPass"
*   Trying <ip-address>:443...
* Connected to <kopia.domain.tld> (<ip-address>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=<kopia.domain.tld>
*  start date: Oct 12 09:06:48 2021 GMT
*  expire date: Jan 10 09:06:47 2022 GMT
*  subjectAltName: host "<kopia.domain.tld>" matched cert's "<kopia.domain.tld>"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Basic with user 'kopia'
* Using Stream ID: 1 (easy handle 0x5565d15ff560)
> GET / HTTP/2
> Host: <kopia.domain.tld>
> authorization: Basic a29waWE6S29waWFQYXNz
> user-agent: curl/7.74.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< accept-ranges: bytes
< content-length: 2378
< content-type: text/html; charset=utf-8
< last-modified: Sat, 16 Oct 2021 10:05:09 GMT
< set-cookie: Kopia-Auth=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrb3BpYS1zZXJ2ZXIiLCJzdWIiOiJrb3BpYSIsImF1ZCI6WyJrb3BpYSJdLCJleHAiOjE2MzQzNzg3NjksIm5iZiI6MTYzNDM3ODY0OSwiaWF0IjoxNjM0Mzc4NzA5LCJqdGkiOiJkYTRhZjBjYS03NGRiLTQ5YjAtODlhYS1jM2M3MzQzZDlhOWMifQ.lVsbiaL9iW7mAHA8ggyx67loOFuQMRTAT6A79TyhDzg; Expires=Sat, 16 Oct 2021 10:06:09 GMT
< date: Sat, 16 Oct 2021 10:05:09 GMT
<
<!doctype html><html lang="en"><head><meta charset="utf-8"/><link rel="icon" href="/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="Web site created using create-react-app"/><link rel="apple-touch-icon" href="logo192.png"/><link rel="manifest" href="/manifest.json"/><title>Kopia UI v0.9.0</title><link href="/static/css/2.c1256375.chunk.css" rel="stylesheet"><link href="/static/css/main.2d1b5133.chunk.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><hr/><p class="version-info">Version v0.9.0 built on Thu Oct 7 15:45:21 UTC 2021 fv-az224-829</p><script>!function(e){function t(t){for(var n,l,i=t[0],f=t[1],a=t[2],c=0,s=[];c<i.length;c++)l=i[c],Object.prototype.hasOwnProperty.call(o,l)&&o[l]&&s.push(o[l][0]),o[l]=0;for(n in f)Object.prototype.hasOwnProperty.call(f,n)&&(e[n]=f[n]);for(p&&p(t);s.length;)s.shift()();return u.push.apply(u,a||[]),r()}function r(){for(var e,t=0;t<u.length;t++){for(var r=u[t],n=!0,i=1;i<r.length;i++){var f=r[i];0!==o[f]&&(n=!1)}n&&(u.splice(t--,1),e=l(l.s=r[0]))}return e}var n={},o={1:0},u=[];function l(t){if(n[t])return n[t].exports;var r=n[t]={i:t,l:!1,exports:{}};return e[t].call(r.exports,r,r.exports,l),r.l=!0,r.exports}l.m=e,l.c=n,l.d=function(e,t,r){l.o(e,t)||Object.defineProperty(e,t,{enumerable:!0,get:r})},l.r=function(e){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(e,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(e,"__esModule",{value:!0})},l.t=function(e,t){if(1&t&&(e=l(e)),8&t)return e;if(4&t&&"object"==typeof e&&e&&e.__esModule)return e;var r=Object.create(null);if(l.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:e}),2&t&&"string"!=typeof e)for(var n in e)l.d(r,n,function(t){return e[t]}.bind(null,n));return r},l.n=function(e){var t=e&&e.__esModule?function(){return e.default}:function(){return e};return l.d(t,"a",t),t},l.o=function(e,t){return Ob* Connection #0 to host <kopia.domain.tld> left intact
ject.prototype.hasOwnProperty.call(e,t)},l.p="/";var i=this.webpackJsonphtmlui=this.webpackJsonphtmlui||[],f=i.push.bind(i);i.push=t,i=i.slice();for(var a=0;a<i.length;a++)t(i[a]);var p=f;r()}([])</script><script src="/static/js/2.f441d67a.chunk.js"></script><script src="/static/js/main.43f0854b.chunk.js"></script></body></html>

Note the server response on the first try (it contains the same output as reported up here):

< HTTP/2 401
< content-type: text/plain; charset=utf-8
< www-authenticate: Basic realm="Kopia"
< x-content-type-options: nosniff
< date: Sat, 16 Oct 2021 10:04:33 GMT
< content-length: 22
<
Missing credentials.

I haven’t implemented the htpasswd configuration part of the repository server documentation.
I will give it a try right away.

No changes. I am lost here.
Regards

For some reason I can not log into the WebUI on Unraid. I have crated a .htpasswd file both from the website, and from terminal and put it in config folder. But the username/password are not accepted.

Also, kopia konsole gives this error
OCI runtime exec failed: exec failed: container_linux.go:367: starting container process caused: exec: "sh": executable file not found in $PATH: unknown

Any help appreciated.

1 Like

Kopia container is just a minimal binary + few extra files and does not come with Linux tooling, in particular shell (/bin/sh). How are you starting the container?

I am using the Kopia container available on the Unraid CA. As there were no instructions about installing any other dependencies, apart from generating a htpasswd file, I was not aware I needed to install anything else.

What should I add to the container?

i am using kopia container on unraid…
i am not using the app available tho…
what issue you are having ?
did you setup the “Post Arguments:” correctly ?

@cyansmoker @ameer

I am trying to configure it in unraid but I always get the same error

kopia: error: path ‘/app/.htpasswd’ does not exist, try --help

To generate the htpasswd file I went to Htpasswd Generator – Create htpasswd - hostingcanada.org used Bcrypt and copy paste the output (for example: admin:$2y$10$D6j8oTdfYEn0jY4ksM5VK.e9bxB9NP7JEfgWSk172VG3KJIZv72CO)
And put the file in the specified route, I create the file with notepad++

server --insecure --htpasswd-file /app/.htpasswd --address 10.10.40.31:51515 --server-username=admin