LZW compression & dictionary

Question

I'm looking into implementing LZW compression in C++, and not sure of the best dictionary implementation.

Hash table made sense, but I don't understand how I would be able to 'reassign' values. If the table gets full, I need to be able to start overwriting previous (oldest) multi-char dictionary entries. Hash table would require me to keep track of these, find it, remove it, and then insert the new one.

Any suggestions?

Is there anything that hinders you from using `std::map` or other standard map implementations? — Desmond Hume, Jul 22 '12 at 15:59
Well, somebody just has to ask "what's wrong with libbzip2"? :-) — Christian Stieber, Jul 22 '12 at 15:59
@ChristianStieber Maybe what's wrong with it is that it doesn't support the extremely fast LZW compression? — Desmond Hume, Jul 22 '12 at 16:01
Would like some practice, not looking for things that make this too simple (STL). — DillPixel, Jul 22 '12 at 16:20

Mark Adler · Answer 1 · 2012-07-22T17:15:18.337

3

The Unix compress utility (source code link) uses double hashing and periodic table clearing.

If you want fast compression and decompression, then there are far better choices than LZW, which is horribly obsolete. You should look at fast, level 1 compression in zlib (probably already on your machine), LZO, and lz4.

There is no reason to write new LZW code other than for instructional or entertainment value. It is only of historical interest. You could also study the compress utility for such instruction and entertainment.

edited Jul 22 '12 at 17:15

answered Jul 22 '12 at 16:50

Mark Adler

101,978
13
118
158

This should be a comment, not an answer. – akappa Jul 22 '12 at 17:01
1

I can't use links the same way, nor can I put paragraphs in a comment. – Mark Adler Jul 22 '12 at 17:03
The reference to the compress source code and its strategy makes this a proper answer, along with a good advice. +1. – akappa Jul 22 '12 at 17:17
2

Wow, The Mark Adler? Cool. – Johan Kotlinski Jan 22 '13 at 22:10

score 2 · Answer 2 · answered Jul 22 '12 at 17:06

You must use two different structures in compression and decompression.

While compressing, you should use a Trie, since you must search the dictionary by content and not by key.

While decompressing, you access the dictionary in the more conventional way, that is, by key. You could then use any associative array structures. like hashtables or even a vector/deque of strings (since your indexes are consecutive natural numbers).

score 1 · Accepted Answer · answered Jul 22 '12 at 16:49

What you're looking for is actually two data structures put together:

A hash table.
A FIFO queue (to discard old table entries)).

You can implement them yourself if you're looking for practice as your comments suggest, or use the stl/sgi/c++11 implementations (unordered_map is the actual hash map, either through sgi or c++11, and a FIFO queue is a doubly linked list, such as std::deque).

The idea is that whenever you want to discard the oldest dictionary entry, you pop the last element in the queue, and then remove it from the hash table as well.

puchu · Answer 4 · 2020-04-17T16:30:59.833

0

You can try 2 dictionaries implemented in lzws:

Linked list, memory usage <= 327 KB.
Sparse array, memory usage <= 33.5 MB.

edited Apr 17 '20 at 16:30

answered Oct 26 '18 at 09:56

puchu

3,294
6
38
62

LZW compression & dictionary

4 Answers4