# Is anybody using LZ4 compression algorithm? **URL:** \ **Category:** General Topics\ **Created:** [April 7, 2024, 8:01pm UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780 "2024-04-07T20:01:11Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [April 7, 2024, 8:01pm UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/1 "2024-04-07T20:01:11Z")
LZ4 algorithm has been deprecated for quite a while and we’re considering removing it from Kopia 0.17.0 (to remove the number of dependencies and tighten our OSS supply chain posture, given recent backdoor in XZ compression library and growing concerns of the OSS ecosystem security as a whole): We’re not expecting a significant number of folks to use LZ4 because it had relatively bad performance compared to others. Please let us know if you’re using LZ4 at scale where removal would cause trouble. If we end up doing it, folks affected would need to use kopia 0.16.1 to migrate their repo to a different compression format before upgrading to v0.17 ---
**Author:** ![iBackup](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/ibackup/32/355_2.png) [@iBackup](https://kopia.discourse.group/u/iBackup)\ **Post date:** [April 8, 2024, 12:40am UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/2 "2024-04-08T00:40:10Z")
> [@jkowalski](#): > > given recent backdoor in XZ compression library AFAIK `xz` isn’t related to `lz4` and `xz` backdoor itself wasn’t sleep in most popular production operations systems. `lz4` is still default compression in ZFS in 2.1.x and while it isn’t most performant, it is still one of lightweight. Also, as far as I know GoLang’s `lz4` isn’t related at all to recently found attack to `xz` due to it’s implementation is in pure Go. Anyway, `xz` attack wasn’t related directly to compression but infrastructure. Removing protocols/algorithms breaks backward compatibility while keeping it, doesn’t make way too much overhead. `lz4` is not outdated legacy and killing it just because of unrelated event, IMO is overkill. It is already implemented, so why to kill it? I know two industrial places where `kopia` uses `lz4` since it shows there optimal benchmarks on limited hardware, so I have no clue how they would be happy if they won’t be able to restore their archives. If it isn’t broken and isn’t vulnerable, I can’t see any reasons to kill `lz4` – Just my 2 cents ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [April 8, 2024, 12:44am UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/3 "2024-04-08T00:44:27Z")
You’re right. I wasn’t trying to imply this particular algorithm or implementation has security issues, just trying to see whether we need to continue supporting it. If you look at performance, it’s not really that fast, allocates a lot of memory compared to others and it’s been deprecated since almost 2 years ago. ```auto Compression Compressed Throughput Allocs Memory Usage ------------------------------------------------------------------------------------------------ 0. zstd-fastest 18.4 MB 6.8 GB/s 812 90.1 MB 1. zstd-better-compression 16.4 MB 5.7 GB/s 780 158.8 MB 2. zstd-best-compression 16.2 MB 5.7 GB/s 791 159.2 MB (deprecated) 3. zstd 18.1 MB 4.6 GB/s 729 157.9 MB 4. deflate-best-compression 18.4 MB 3.7 GB/s 864 650.6 KB 5. deflate-default 19.7 MB 3.5 GB/s 944 644.2 KB 6. deflate-best-speed 20.6 MB 3.4 GB/s 496 641.4 KB 7. pgzip-best-compression 18.4 MB 3 GB/s 3000 67.9 MB 8. pgzip 19.7 MB 3 GB/s 3332 67.9 MB 9. pgzip-best-speed 20.6 MB 2.8 GB/s 2861 67.9 MB 10. s2-default 35.1 MB 1.8 GB/s 122 4.3 GB 11. gzip-best-speed 21.6 MB 1.7 GB/s 137390 4.3 GB 12. s2-better 33.7 MB 1.5 GB/s 125 4.3 GB 13. lz4 33.9 MB 1.4 GB/s 108 4.4 GB (deprecated) 14. s2-parallel-4 35.1 MB 1.2 GB/s 110 4.3 GB 15. s2-parallel-8 35.1 MB 1.1 GB/s 126 4.3 GB 16. gzip-best-compression 18.5 MB 1 GB/s 149055 4.3 GB 17. gzip 19.3 MB 683.2 MB/s 142871 4.3 GB ``` BTW. Another option would be to keep it as opt-in feature when building Kopia, this way folks who rely on it would have a choice of including it, but official binaries would not. ---
**Author:** ![iBackup](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/ibackup/32/355_2.png) [@iBackup](https://kopia.discourse.group/u/iBackup)\ **Post date:** [April 8, 2024, 1:49am UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/4 "2024-04-08T01:49:43Z")
> [@jkowalski](#): > > just trying to see whether we need to continue supporting it. If it would be me, I would support it as long as ZFS will do it. > [@jkowalski](#): > > If you look at performance, it’s not really that fast I know, as about me I do in most cases `s2-default`, but take a look (ssh-ed to one pretty old internal email server and tried it on one email with attachments) ```auto Compression Compressed Throughput Allocs Usage ------------------------------------------------------------------------------------------------ 0. s2-parallel-4 106.9 KB 626.3 MB/s 2644 2.4 MB, --compression=s2-parallel-4 1. s2-default 106.9 KB 623.2 MB/s 2677 2.4 MB, --compression=s2-default 2. s2-parallel-8 106.9 KB 620.4 MB/s 2586 2.4 MB, --compression=s2-parallel-8 3. s2-better 100.9 KB 379.4 MB/s 2607 2.4 MB, --compression=s2-better 4. zstd-fastest 80.7 KB 321.1 MB/s 7423 10.2 MB, --compression=zstd-fastest 5. zstd 77.6 KB 223.6 MB/s 4800 20.3 MB, --compression=zstd 6. deflate-best-speed 110.8 KB 217.5 MB/s 29 1.1 MB, --compression=deflate-best-speed 7. lz4 111.8 KB 182.7 MB/s 1929 3.6 GB (deprecated), --compression=lz4 8. zstd-better-compression 72.2 KB 154.3 MB/s 4866 23 MB, --compression=zstd-better-compression 9. gzip-best-speed 114.1 KB 150.1 MB/s 35 1.5 MB, --compression=gzip-best-speed 10. deflate-default 86.3 KB 139.5 MB/s 29 1.3 MB, --compression=deflate-default 11. pgzip-best-speed 110.8 KB 120.9 MB/s 10536 1.1 GB, --compression=pgzip-best-speed 12. pgzip 86.3 KB 96.8 MB/s 10187 1.1 GB, --compression=pgzip 13. gzip 82.5 KB 67 MB/s 33 1.1 MB, --compression=gzip 14. gzip-best-compression 82.3 KB 58.6 MB/s 33 1.1 MB, --compression=gzip-best-compression 15. deflate-best-compression 82.1 KB 56.1 MB/s 30 1.4 MB, --compression=deflate-best-compression 16. pgzip-best-compression 82.1 KB 46.1 MB/s 10470 1.2 GB, --compression=pgzip-best-compression 17. zstd-best-compression 68.5 KB 35.2 MB/s 4829 54.6 MB (deprecated), --compression=zstd-best-compression ``` `lz4` defiantly isn’t great , but isn’t worst too. I don’t use `lz4` for sure, but my point is, - if it isn’t broken or vulnerable and isn’t abandoned (last change in `lz4` was just 5 months ago), then why to get rid of it? It doesn’t take a lot of space and is not requires a special handling to support in compare to a bunch of clouds. IMO in case of backup software - “then more than better” as far as it doesn’t creates a problems > [@jkowalski](#): > > Another option would be to keep it as opt-in feature when building Kopia, this way folks who rely on it would have a choice of including it, but official binaries would not. At least it would be much better than just drop support with no reason and make some1 rebuild backup from scratch. I wish such **official** future would exists to get rid of unused clouds (like Azure, GCP, AWS…) since those makes `kopia` really fat while most of the time the only one out of premises cloud is in use. ---
**Author:** ![kapitainsky](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/kapitainsky/32/535_2.png) [@kapitainsky](https://kopia.discourse.group/u/kapitainsky)\ **Post date:** [April 8, 2024, 5:05am UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/5 "2024-04-08T05:05:02Z")
I stand with @iBackup comments. It is not good idea to remove it. Not sure where information about LZ4 being deprecated is from. It is still in use - not only as a default in ZFS. ---
**Author:** ![jkowalski](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/jkowalski/32/3_2.png) [@jkowalski](https://kopia.discourse.group/u/jkowalski)\ **Post date:** [April 9, 2024, 12:06am UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/6 "2024-04-09T00:06:44Z")
To make sure it’s clear what I mean: the algorithm is deprecated within Kopia not anywhere else, and it has to do with particular implementation we’re using, not the format itself. Kopia does currently not prevent you from using lz4 (we only hide it when you run benchmark by default), we just recommend folks don’t use it for new repos. ---
**Author:** ![IcePic](https://avatars.discourse-cdn.com/v4/letter/i/74df32/32.png) [@IcePic](https://kopia.discourse.group/u/IcePic)\ **Post date:** [April 9, 2024, 5:03pm UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/7 "2024-04-09T17:03:11Z")
I would not mind if it goes. s2 is clearly the better speed-compressor and zstd handles where ratio is important. ---
**Author:** ![nh2](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/nh2/32/171_2.png) [@nh2](https://kopia.discourse.group/u/nh2)\ **Post date:** [May 8, 2024, 12:36pm UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/8 "2024-05-08T12:36:45Z")
I’d like to point out that the mentioned problems (high memory allocation and low throughput) are likely due to the pure-Go implementation. The original `lz4` (made by the same author as `zstd`), remains the highest-throughput and lowest-memory compressor on the planet (to my knowledge), see e.g. [official README](https://github.com/lz4/lz4/blob/c1501a248f3d11506f51c211c207ab086fbb8432/README.md#benchmarks). As shown there, `lz4` compresses 1.5x as fast and decompresses 4x as fast than `zstd -1`. `zstd` does not not replace `lz4`; they are different compressors for different tasks. I use `lz4` over `zstd` on all fast storage systems, and it makes sense to use `zstd` as a default on-the-fly compressor for e.g. ZFS or Ceph, because it has so low CPU and memory overhead that there is almost no situation in which LZ4 does NOT make sense. That said, I would not use the pure Go `lz4` in Kopia if its implementation is slower than `zstd`. But I likely would use `lz4` in Kopia if it was as fast as the original `lz4`. ---
**Author:** ![dimejo](https://yyz2.discourse-cdn.com/free1/user_avatar/kopia.discourse.group/dimejo/32/287_2.png) [@dimejo](https://kopia.discourse.group/u/dimejo)\ **Post date:** [May 9, 2024, 9:58pm UTC](https://kopia.discourse.group/t/is-anybody-using-lz4-compression-algorithm/2780/9 "2024-05-09T21:58:04Z")
> [@nh2](#): > > The original `lz4` (made by the same author as `zstd`), remains the highest-throughput and lowest-memory compressor on the planet (to my knowledge), see e.g. [official README](https://github.com/lz4/lz4/blob/c1501a248f3d11506f51c211c207ab086fbb8432/README.md#benchmarks). I downloaded the Silesia Corpus files and ran my own test. LZ4 and Zstandard both with `-1` compression level were equally fast (0.4 vs 0.46 seconds), but Zstandard’s compression ratio (97 vs 71 MB) was a lot better. zstd run with `--fast=3` was even faster (0.38 seconds) than lz4 while preserving a better compression ratio (91 MB). The results in the [zstd benchmark](https://facebook.github.io/zstd/) show similiar numbers as the lz4 benchmark, but I can’t confirm this with my tests. Maybe it’s because of the lzbench tool they are using? Have you ever tested this yourself? > [@nh2](#): > > almost no situation in which LZ4 does NOT make sense LZ4 doesn’t make sense if you care about compression ratio but it is awesome if you need fast decompression.