How to put nginx in front?

Support
1

I’m trying to put nginx in front. I generated the cert with kopia and then added the crl and key to nginx. When I try and connect from the client I get…

# kopia repository connect server --url https://myurl:443   --server-cert-fingerprint ddcafd58b5943e015a2e9f4a4df849f1f2bf4b82346b4e416   
Connecting to server 'https://myurl:443' as 'user1@server1'...
Enter password to open repository:

failed to open repository: unable to establish session for purpose=: error establishing session: Session(): rpc error: code = Unavailable desc = connection closed
kopia: error: error connecting to API server: unable to establish session for purpose=: error establishing session: Session(): rpc error: code = Unavailable desc = connection closed, try --help
#

This is what I see in the nginx access log.

myurl:443 6.29.1.1 - - [25/Apr/2021:16:45:09 -0700] "PRI * HTTP/2.0" 400 173 "-" "-"

nginx config

server {
  listen 443;
  server_name myurl;

    ssl    on;
 

    ssl_certificate_key /var/lib/kopia/kopia-cert.key;
    ssl_certificate /var/lib/kopia/kopia-cert.cert;

  client_max_body_size 0;  # unlimited

  location / {
   proxy_pass https://localhost:51515;
  }
}
2

Kopia server uses gRPC which is based on http/2. Not sure exactly how this impacts nginx reverse proxying.

3

Thanks. Would not have figured this out without your tip. The below passes some basic tests.


server {
  listen 443 ssl http2;
  server_name myurl;

    ssl    on;
    ssl_certificate_key /var/lib/kopia/kopia-cert.key;
    ssl_certificate /var/lib/kopia/kopia-cert.cert;

  client_max_body_size 0;  # unlimited

  location / {
   grpc_pass grpcs://localhost:51515;
  }
}
4

BTW, would be great to document this on kopia.io, I’ll be happy to accept a PR:

The contents are here: kopia/site/content/docs at master · kopia/kopia · GitHub

5

I wanted to share my findings on using nginx:

  • the provided nginx example from @bbrendon works fine
  • you have to use grpcs. grpc does not work. This means the kopia server has to be started with a certificate (–insecure does not work). It can be a self-signed certificate, that does not matter.
  • you have to use nginx >= 1.16; with nginx 1.14 I get the following error
    upstream sent frame for closed stream 1 while reading upstream which is fixed with a recent nginx version (see here: #1792 (grpc module handles RST_STREAM(NO_ERROR) improperly on closed streams) – nginx)
    I tested and I got no error with nginx 1.18
  • You can use https://yourdomain.com:443 in kopia / kopia-ui to connect to the server
  • I did a quick test with backup, restore and policy modification in kopia-ui; everything worked
  • from time to time I can see an error in the logs:
    [error] 31#31: *15 upstream timed out (110: Connection timed out) while reading upstream... request: "POST /kopia_repository.KopiaRepository/Session HTTP/2.0
    I could not figure out why this occurs, but I couldn’t find anything not working either. Increasing proxy timeouts did not help - the error occurred before the timeout was reached.
6

Hi there,
I am having trouble using HAProxy in front (as detailed elsewhere), and just gave a try with Nginx (version 1.18.0).
I start the server with self-signed certificates, but the connection to the repository server using command line doesn’t work.

Using client Kopia command line:

❯ kopia repository connect server --url=https://<kopia.domain.tld>:443 --server-cert-fingerprint=<fingerprint> --log-file=kopia.log --log-level=debug --override-username=<username> --override-hostname=<hostname> --password=<password>

Nginx on the server returns:

2021/10/21 19:34:34 [error] 59303#59303: *71 upstream sent frame for closed stream 1 while reading upstream, client: <client-ip>, server: <kopia.domain.tld>, request: "POST /kopia_repository.KopiaRepository/Session HTTP/2.0", upstream: "grpcs://0.0.0.0:51515", host: "<kopia.domain.tld>:443"

Client Kopia log returns:

❯ cat kopia.log
2021-10-21 19:34:33.168 I [logger.go:244] Connecting to server 'https://<kopia.domain.tld>:443' as '<user@hostname>'...
2021-10-21 19:34:33.168 D [logger.go:254] Creating cache directory '/home/<$USER>/.cache/kopia/1b0e0fd7e3249467' with max size 5242880000
2021-10-21 19:34:33.374 D [logger.go:254] establishing new GRPC streaming session (purpose=)
2021-10-21 19:34:33.702 D [logger.go:254] GRPC stream read loop terminated with rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR
2021-10-21 19:34:33.702 E [logger.go:214] failed to open repository: unable to establish session for purpose=: error establishing session: unable to initialize session: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR: EOF

I have added the user <username>@<hostname> serverside.

Also, curl -v https://<kopia.domain.tld>:443 -u <server-username>:<server-password> shows correct TLS handshakes, as with using HAProxy and reported here.

Any hints?