What is zstd-best-compression, and can I customize?

General Topics
1

I’ve been playing with zstd lately, and find excellent compression using -22 –ultra –long and would like to use this with Kopia, if possible.

Storage space is costing me, so I’m looking to compress maximally at the expense of time.

What is the equivalent of zstd-best-compression? I couldn’t find it in the docs. I presume it is -19 since that’s the highest “non-special” level in zstd by default?

Thanks!

EDIT: On customizing, is there a way to have Kopia use an “external” compression utility, similar to how tar has this?

2

Not that I am aware of. What purpose would that serve? Kopias comrpession options cover all possible ranges, so why try to rely on an external one?

And “best” zstd compression is always subjective. Have you tried to run kopia benchmark on some on your existing data?

3

Kopia uses https://github.com/klauspost/compress package for zstd compression. And its implementation of different compression levels (inherited by kopia) is as follows:

The "Fastest" compression ratio is roughly equivalent to zstd level 1.
The "Default" compression ratio is roughly equivalent to zstd level 3 (default).
The "Better" compression ratio is roughly equivalent to zstd level 7.
The "Best" compression ratio is roughly equivalent to zstd level 11.

Changing it would require using different more flexible zstd package and pluging it into kopia. Of course such implemetation would have to be backward compatible. All is open source so if somebody really needs it just have to do it.

4

I Have run kopia benchmarks on my data.

I ran a test, where I created a single snapshot of a dataset using kopia, then used tar –use-compress-program ‘zstd -22 –ultra –long=30’ to compare the two. The original data was 466 MB, the kopia repo was 323 MB on disk, and the tar.zst was 159 MB. Pretty big difference, I’d like to use Kopia with the best possible compression I can squeeze out.

EDIT: Included the original data size, for context.

5

The best available today as already mentioned is zstd 11. If you need something better than either compress your data up front or give it a go and suggest PR.

6

It could also be, that the dedup algorithm, which actually chops your data int chunks, has an effect on the overall compressratio…