Kopia server taking too much ram->swapping

Support
1

I have and old server with 6Gb of ram running a kopia server (and other services…).
Every now and then kopia starts taking 20% of ram and the server starts swapping. I tried to stop other services to allow more ram, but it is useless, the only solution is restarting kopia service.
Is there any CLI command to see what is kopia doing? I tried accessing web interface, but server is barely responding to a SSH session.

Thanks

2

Can you try with the latest v0.11.0 - there have been a bunch of improvements to reduce the memory usage in that version.

3

Ok, I just upgraded my version using apt.
Maybe there is a parameter I could add to my kopia.service (systemd) to make it more verbose?

4

Checking the changelog I see zstd-best-compression is not recommended, and it is the algorithm I was using… which one is recommended and how should I proceed to change it? I have:

  • multiple clients (family PCs) connecting to my server
  • server is low on CPU, but I prefer “best compression possible”, I thing for long retention it pays back.
  • the storage is a S3 (wasabi, no egress fee, only storage billed), the bucket is ~1Tb

Just changing the policy in the server will affect new upload and repacking? will deduplication still work?

5

You can see some benchmarking statistics for the different compression algorithm at Compression | Kopia. Since you are low on RAM, I would pick one that uses less memory.

ztd is the algorithm that provides one of the best compression ratios with very good speed, but RAM usage is on the higher side. Looking at Compression | Kopia, gzip uses very little RAM but it is very slow. pgzip is the middle ground – less RAM than ztd but more than gzip with decent compression ratios and faster speed than both. May I suggest you try pgzip to see if your server can handle that? If not, change it gzip.

Changing the policy now would change compression on all future uploads. Kopia will not go back and recompress already uploaded files. Deduplication should still work.

6

My server is a dual core machine, so I don’t think there should be much of a difference between gzip and pgzip.
I will play with gzip, and change it whenever I upgrade the server.

7

If it might help, - I having one server with 4Gb of RAM and just zstd works fine ( I stay away from “zstd-best-compression”) even so there working MySQL, Web server + email server (postfix+dovecot+spamassasin+antivirus+DKIM+DCC…)

8

No, it don’t work fine, it worked a few months Ok, but today’s latest maintenance got failed due to out of memory because of zstd. I rerun kopia benchmark compression and it looks like s2 is the fastest and much less hungry.

It looks like pgzip usage is more than zstd in my tests (320kb data file)

     Compression                Compressed   Throughput   Allocs   Usage
------------------------------------------------------------------------------------------------
  0. s2-parallel-8              104.5 KiB    606.2 MiB/s  2159     2.2 MiB, --compression=s2-parallel-8
  1. s2-parallel-4              104.5 KiB    605.8 MiB/s  2187     2.2 MiB, --compression=s2-parallel-4
  2. s2-default                 104.5 KiB    603.1 MiB/s  2195     2.2 MiB, --compression=s2-default
  3. s2-better                  99.6 KiB     435.2 MiB/s  2263     2.2 MiB, --compression=s2-better
  4. zstd-fastest               78.8 KiB     298.9 MiB/s  7368     9.7 MiB, --compression=zstd-fastest
  5. zstd                       75.8 KiB     206.3 MiB/s  4879     19.3 MiB, --compression=zstd
  6. deflate-best-speed         98.2 KiB     205.2 MiB/s  29       1.1 MiB, --compression=deflate-best-speed
  7. zstd-better-compression    71 KiB       147.9 MiB/s  4872     37.8 MiB, --compression=zstd-better-compression
  8. gzip-best-speed            111.5 KiB    144.3 MiB/s  35       1.5 MiB, --compression=gzip-best-speed
  9. deflate-default            84.7 KiB     132.8 MiB/s  29       1.3 MiB, --compression=deflate-default
 10. lz4                        109.2 KiB    122.4 MiB/s  2135     3.4 GiB, --compression=lz4
 11. pgzip-best-speed           98.2 KiB     109.2 MiB/s  11456    1.1 GiB, --compression=pgzip-best-speed
 12. pgzip                      84.7 KiB     77.7 MiB/s   11315    1.1 GiB, --compression=pgzip
 13. gzip                       80.6 KiB     63.7 MiB/s   33       1.1 MiB, --compression=gzip
 14. gzip-best-compression      80.4 KiB     56.1 MiB/s   33       1.1 MiB, --compression=gzip-best-compression
 15. deflate-best-compression   80.2 KiB     46 MiB/s     30       1.4 MiB, --compression=deflate-best-compression
 16. pgzip-best-compression     80.2 KiB     38.2 MiB/s   11807    1.2 GiB, --compression=pgzip-best-compression
 17. zstd-best-compression      68.3 KiB     22.1 MiB/s   4819     100 MiB, --compression=zstd-best-compression
9

Yes, you are correct. I was focusing on large files, but RAM usage on small files for pgzip is quite high.

10

It looks like it isn’t an issue with zstd. Replaced 0.11 back to kopia 0.10.7 and it finished maintenance successfully, but I will keep s2 compression from now on anyway.

11

Could the issue you are running into be related to Regression: kopia maintenance failed on v0.11.2 due to out of memory · Issue #2161 · kopia/kopia · GitHub?

12

I’m coming back to try kopia again after almost a year and I’m currently creating a new snapshot of a folder that has 675,077 files and as far as I’ve seen kopia uses up to 20GB of RAM (I’ve had to update that number 3 times in the making of this comment because it first was 16 and then 19), sometimes I have seen it go down to 8 or 9 GB but I think that is due to memory compression because that process’ RAM usage goes up too. Is this a currently known problem?
I’m using the default parallel file reads which should be 16 and using zstd-better-compression, CPU usage is pretty low, only thing is the RAM usage, it’s too much. restic uses a lot too but it goes to 7GB or less.