3

I am confused about the interpretation of the minimum description length of an alphabet of two symbols.

To be more concrete, suppose that we want to encode a binary string where 1's occur with probability 0.80; for instance, here is a string of length 40, with 32 1's and 8 0's:

1 1 0 1 1 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 0 1 1 1 1 0 1 1 0 0 1

Following standard MDL analysis, we can encode this string using prefix codes (like Huffman's) and the code of encoding this string would be (-log(0.8) * 32 - log(0.2) * 8), which is lower than duplicating the string without any encoding.

Intuitively, it is "cheaper" to encode this string than some string where 1's and 0's occur with equal probability. However, in practice, I don't see why this would be the case. At the very least, we need one bit to distinguish between 1's and 0's. I don't see how prefix codes could do better than just writing the binary string without encoding.

Can someone help me clarify this, please?

Jose Cadena
  • 65
  • 4

1 Answers1

3

I don't see how prefix codes could do better than just writing the binary string without encoding.

You can't with prefix codes, unless you combine bits to make more symbols. For example, if you code every two bits, you now have four symbols with probabilities 0.64, 0.16, 0.16, and 0.04. That would be coded with 0, 10, 110, 111. That gives an average of 1.56 bits per symbol, or 0.7800 bits per original bit. We're getting a little closer to the optimal 0.7219 bits per bit (-0.2 log20.2 - 0.8 log20.8).

Do that for three-bit groupings, and you get 0.7280 bits per bit. Surprisingly close to the optimum. In this case, the code lengths just happen to group really nicely with the probabilities. The code is 1 bit (0) for the symbol with probability 0.512, 3 bits (100, 101, 110) for the three symbols with probability 0.128, and 5 bits (11100, 11101, 11110, 11111) for both the three symbols with probability 0.032 and the one symbol with probability 0.008.

You can keep going and get asymptotically close to the optimal 0.7219 bits per bit. Though it becomes more inefficient in time and space for larger groupings. The Pareto Front turns out to be at multiples of three bits up through 15. 6 bits gives 0.7252 bits per bit, 9 gives 0.7251, 12 is 0.7250, and 15 is 0.7249. The approach is monumentally slow, where you need to go to 28 bits to get to 0.7221. So you might as well stop at 6. Or even just 3 is pretty good.

Alternatively you can use something other than prefix coding, such as arithmetic coding, range coding, or asymmetric numeral system coding. They effectively use fractional bits for each symbol.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • Thank you for your help. That makes sense. – Jose Cadena Sep 18 '15 at 20:05
  • This was an incredibly helpful post / answer. Can you expand your answer to show how the encoding of Jose's original bitstring would look, given the new way to encode the bitstring by 2bit chunks? I'm trying to figure out how one would decode such a transformed bitstring, into the original bitstring Jose provided. You would have to encode the mapping from 00 to 0, and 01 to 10, 10 to 110, and 00 to 111 in the actual encoding, correct? But wouldn't that bitstring be longer than the original, then? – lrthistlethwaite Jan 11 '19 at 04:29
  • `0100101100000000100110001011010` – Mark Adler Jan 11 '19 at 07:05