Compression Algorithm for Small Amounts of Data

Question

I have a server-side program that generates JSON for a client. A few of my colleagues have suggested using zip/gzip compression in order to reduce the amount of data that sending over the wire. However, when tested against one of my average JSON messages, it they both actually increased the amount of data being sent. It wasn't until I sent an unusually large response that the zipping kicked in and was useful.

So I started poking around stackoverflow, and I eventually found LZO, which, when tested, did exactly what I wanted it to do. However, I can't seem to find documentation of the run time of the algorithm, and I'm not quite good enough to sit down with the code and figure it out myself :)

tl;dr? RUN TIME OF LZO?

[Java, Javascript, Objective-C, Python] <-- Current list of languages. More may come later for one reason or another. — Patrick Perini, Jul 15 '11 at 12:41
It's obviously no worse than O(n log n), or it'd be unusable for large texts. Have you considered deflate, though? It's supported directly by most HTTP clients and servers. — Nick Johnson, Jul 15 '11 at 13:03

score 8 · Accepted Answer · answered Jul 15 '11 at 13:03

I'm going to ignore your question about the runtime of LZO (answer: almost certainly fast enough) and discuss the underlying problem.

You are exchanging JSON data structures over the wire and want to reduce your bandwidth. At the moment you are considering general-purpose compression algorithms like DEFLATE and LZO. However, any compression algorithm based on Lempel-Ziv techniques works best on large amounts of data. These algorithms work by building up a dictionary of frequently occurring sequences of data, so that they can encode a reference to the dictionary instead of the whole sequence when it repeats. The bigger the dictionary, the better the compression ratio. For very small amounts of data, like individual data packets, the technique is useless: there isn't time to build up the dictionary, and there isn't time for lots of repeats to appear.

If you are using JSON to encode a wire protocol, then your packets are very likely stereotyped, with similar structures and a small number of common keys. So I suggest investigating Google's Protocol Buffers which are designed specifically for this use case.

score 4 · Answer 2 · answered Jul 15 '11 at 13:23

Seconding the suggestion to avoid LZO and any other type of generic/binary-data compression algorithm.

Your other options are basically:

The best choice depends on your server/language setup, your speed-vs-compression requirements, and personal preference. I'd probably go with MessagePack myself, but you won't go wrong with Protocol Buffers either.

Compression Algorithm for Small Amounts of Data

2 Answers2