Hellishly Slow Level 13 Deflate Compression
kirill.korins.ky41 points by zX41ZdbW 5 days ago
41 points by zX41ZdbW 5 days ago
As someone currently exploring grid searches of encodings + compressor combos, and currently looking at neural compressors that reduce size almost half that of a traditional compressor yet take order from ms -> minutes to operate in either direction, I appreciate a good compression post!
I love it. So much in computers is trade offs and this was a fun read exploring it.
It would be interesting to see some economics of what 8,000% increase in encoding time takes to make that money back in terms of storage or bandwidth. I also wonder how brotli/lzma would compare here. Are there some obscene modes on those that had similar results?
zstd has higher level modes. Default is -3. I saw a good tradeoff between compression speed and ratio up to -9 or so. From -20 to -22 it will use much more memory and IIRC can have downstream effects on decompression speed. I'm using -9 for my container registry and plan to recompress at a higher level for commonly accessed base layers, as well as give customers a button that lets them pay a bit more to do it themselves.
To be a little pedantic, the usual zstd levels are positive integers (1-22 default 3). The negative integers denote "fast" modes with worse compression (there are only a few of these).
I also wonder how brotli/lzma would compare here.
Far better, just like anything else based on arithmetic coding. The main distinction here is that the output can still be decompressed with a standard Inflate implementation.
reminds me of the x264 “placebo” encoder setting
There is also zopfli and it's decadent ECT that allow for more extreme tradeoffs.
So, what’s the effect on memory usage?
And for decompression, the effect on memory usage and timings?
For decompression, nothing changes because DEFLATE is asymmetric; compressor can spend however much time to optimize the compressed stream independently from decompressor.
It's interesting to see just how far Deflate can be taken, and to know that even after decades there is still some (admittedly tiny) room for improvement. Optimal LZ is well-known, and so is static Huffman, but their combination creates some additional inefficiencies(opportunities).
...and of course it's written by someone with a Russian name, and has that characteristic style common to many other articles about data compression.