19

Does anyone know of a free (non-GPL), decently performing compression library that supports packet oriented compression in C/C++?

With packet oriented, I mean the kind of feature QuickLZ (GPL) has, where multiple packets of a stream can be compressed and decompressed individually while a history is being maintained across packets to achieve sensible compression.

I'd favor compression ratio over CPU usage as long as the CPU usage isn't ridiculous, but I've had a hard time finding this feature at all, so anything is of interest.

porgarmingduod
  • 7,668
  • 10
  • 50
  • 83
  • What is it good for to have packet oriented compression? When you maintain dictionary from previous packets you need to decompress all previous packets anyway. – Juraj Blaho May 12 '11 at 10:11
  • Of course you need to decompress "all previous packets". But many applications are interested in each individual packet by themselves, as they arrive. The compression algorithm, on the other hand, is interested in the data as a stream in order to detect entropy, and thus want to build up it's state over time. The goal should be obvious enough: To allow the sender to transmit packets containing verbose (i.e low entropy) data, and have a compression algorithm automatically minimize the actual bits sent. – porgarmingduod May 12 '11 at 12:27
  • The purpose of packet oriented compression is still not obvious to me. It should be easy to partition the decompressed stream into original packets if you need to do so. – Juraj Blaho May 12 '11 at 13:55
  • 2
    I think the OP wants a guarantee that he'll get small chunks of decompressed data in a timely manner, instead of having to wait for full decompression before getting any data. – Michael Kohne May 12 '11 at 14:30
  • What do you mean by "free (non-GPL)" ? Does BSD license fit the bill ? – Cyan Oct 06 '11 at 11:50
  • Do you have a BSD licensed suggestion, or is this going off topic? For me personally, "free" means the ability to use the product of the code (i.e. binaries) without hassle. Sometimes GPL is bothersome. The BSD clause isn't that much bother. MIT is no bother at all. – porgarmingduod Oct 06 '11 at 14:03
  • Did you ever find a good solution for this? – Alastair Maw Feb 26 '13 at 17:54
  • Yes, I did, it just took me a while. The answer which I just now accepted is actually very decent. – porgarmingduod Feb 27 '13 at 12:15

5 Answers5

9

zlib's main deflate() function takes a flush parameter, which allows various different flushing modes. If you pass Z_SYNC_FLUSH at the end of each packet, that should produce the desired effect.

The details are explained in the zLib manual.

bzip2 has flushing functionality as well, which might let you do this kind of thing. See http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#bzCompress

pmdj
  • 22,018
  • 3
  • 52
  • 103
  • Yep. And zlib license is NOT copyleft. It's more akin to BSD or MIT that GPL. Basically, you just can't 1) remove the copyright notice when redistributing **source** or 2) Modify it, and still call it ZLib. – Brian McFarland Oct 04 '11 at 15:11
  • This is actually a fairly good way to do it; `Z_SYNC_FLUSH` retains the compression state and simply writes enough data so that the receiving end can read what has been written so far. I guess accepting after a year and a half is still better than never :) – porgarmingduod Feb 27 '13 at 12:14
  • I'd also like to add at this point that http://code.google.com/p/lzham/ (as of alpha8) supports zlib style flushing, and can be used in the same manner with different performance characteristics. – porgarmingduod Apr 09 '14 at 10:54
5

Google's Snappy may be a good option, if you need speed more than compression and are just looking to save a moderate amount of space.

Alternatively, Ilia Muraviev put a small piece of compression code called BALZ in public domain some time ago. It is quite decent for many kinds of data.

Both of these support stream flushing and independent state variables to do multiple, concurrent streams across packets.

Seth
  • 2,683
  • 18
  • 18
  • Do they? I can't find that in Snappy - looks like it just compresses with a stateless function. BALZ is command line, not a library, and looks like it just streams stdin in chunks, not supporting packet-type stuff at all? – Alastair Maw Feb 26 '13 at 17:50
2

Google's new SPDY protocol uses zlib to compress individual messages, and maintains the zlib state for the life of the connection to achieve better compression. I don't think there's a standalone library that handles this behavior exactly, but there are several open-source implementations of SPDY that could show you how it's done.

Bob
  • 3,301
  • 1
  • 16
  • 11
  • Interesting, and good thinking to suggest it for this question. Do you happen to know how small messages can be sensibly sent? – porgarmingduod Oct 06 '11 at 20:22
1

The public domain Crush algorithm by Ilia Muraviev has similar performance and compression ratio as QuickLZ has, Crush being a bit more powerful. The algorithms are conceptually similar too, Crush containing a bit more tricks.
The BALZ algorithm that was already mentioned earlier is also by Ilia Muraviev.
See http://compressme.net/

Roland Pihlakas
  • 4,246
  • 2
  • 43
  • 64
0

may be you could use lzma compression SDK, it's written and placed in the public domain by Igor Pavlov.

And since it can compress stream files, and has memory to memory compression I think it's possible to compress packet stream (may be with some changes) but not sure.

chedi
  • 298
  • 3
  • 12
  • 1
    There are a lot of open source, free to use compression libraries. Compression is also highly technical procedure. Simply assuming lzma, which is specifically written for the highest possible performance on large datasets, could be easily modified to work with small packets of data (a quite different problem) is going to require more than "maybe" and "not sure". – porgarmingduod Oct 03 '11 at 22:41