searching bit-field templates (codebooks)

Question

I've got a bunch of 8-bit values in a codebook (about 200 of them).

My program will be generating an 8-bit value in response to input, and I need to find all (or even the first is helpful) of the matches in the codebook that have the same bits set. The bits that are unset don't matter.

Can you think of an optimal way to a) store and b) search the codebook to find all matches? I have a standard linear search in place but of course it's pretty inefficient.

Many thanks...

akevan

add a tag to indicate what programming language you would like to use. Many people only scan postings only by their specialiality tags. Good Luck. — shellter, Jun 07 '11 at 23:45

Seth Robertson · Answer 1 · 2011-06-29T18:20:36.787

2

Of course, while not very space efficient, you can of course precompute the matches for all 256 bit patterns. You would have an array of 256 lists, each list would contain each code in the codebook with those bits set.

You can get the first match in 256 bytes (11 words of memory).

initialization:

u_int8_t bitpatterns[256];
memset(bitpatterns,0,sizeof(bitpatterns));

for(x=sizeof(codebook)-1;x>=0;x--)
  for(y=0;y<256;y++)
    if (y&codebook[x] == y)
      bitpatterns[y] = x;

Lookup:

codeword = codebook[bitpatterns[input]];

edited Jun 29 '11 at 18:20

answered Jun 29 '11 at 17:24

Seth Robertson

30,608
7
64
57

Unfortunately I can't do this as it will consume too much memory. This is one small part of an algorithm going into a device with 10K total memory... – akevan Jun 29 '11 at 17:37
@akevan: How much space can you afford? Do you know anything about the distribution of bits? (Are they random? Standard distribution? Are there more 1 bits set than 0 bits in the codebook? In the pattern?) Though...if you only want *a* answer (not all answers) you can do that with a precomputed array of 256 bytes (or instead of bytes, offsets (bytes) into the array of codewords for the first matching codeword) – Seth Robertson Jun 29 '11 at 17:57
Good questions... they're random in distribution. I guess the fact that the codebook contains nearly all bit-patterns is helpful (and also points towards a random distribution!), but I'm still trying to figure out how to take advantage of it. I have about 500 words of space to use (24-bit words). – akevan Jun 29 '11 at 18:08
@akevan: Well, as I said, you can get the first codebook match in 11 words of memory. u_int8_t bitpattern[256]; …; codebook[bitpattern[input]]; – Seth Robertson Jun 29 '11 at 18:17
@Seth: Thanks for the info, very helpful. Will continue to work on this - tried to "up" your answer but I can't because I have no reputation hahah... thanks though! – akevan Jun 29 '11 at 18:21
@akevan: Could you store **only** the precalculated match-table, or do you need to store the actual codes? – larsmoa Jun 29 '11 at 18:22
@akevan: Every 256 bytes gives you another match. What you could do is have a second or third "normal" match giving an answer. On the third match, if the value is 256 (or some other illegal value) that would signal that you must do a linear search to generate the full list. Of course, you could reverse that and make that the *first* match. 1, 2, or 3 entry matches you use the accelerator, 4+ entry matches you know you need to do a linear search (and perhaps the second entry's number would tell you what offset to start the search at). – Seth Robertson Jun 29 '11 at 18:26
@akevan: Last suggestion, 384 bytes gives you: first match, last match, middle match. If middle match is -2 you need to search the list between first and last for other matches (there will be at least two, the negative number could tell you the exact number for modest (~50) numbers of matches), if it is -1 there were only two matches (if last is -1 there was only one match, if first is -1 there were no matches). – Seth Robertson Jun 29 '11 at 18:53

larsmoa · Answer 2 · 2011-06-29T19:23:29.993

1

One optimization could be to store your codes in different buckets depending on how many bits are set. When you're looking up codes you would just have to look through 1/2 of the codes (in average, if codes are distributed uniformly). This is a very simple optimization, but the complexity of the algorithm remains the same (O(n)). Sorting a single array based on the number of bits set would enable you to do similar optimizations without having to store the codes in buckets.

Sidenote: I think 200 is a very small number and I don't think you would see much change in performance from a linear approach no matter how you optimize this unless you do a lot of lookups. But I'm guessing that's not the point of this exercise...

edited Jun 29 '11 at 19:23

answered Jun 29 '11 at 17:06

larsmoa

12,604
8
62
85

1

Hmmm... I like the idea, but I'd still have to search all the buckets. Given the requirement for a match (the pattern must have the same bits set as the input), if I have a input of 0001 1100 (3 bits set), it would match 0011 1100 (4 bits set), 0111 1100 (5 bits), etc. – akevan Jun 29 '11 at 17:41
@akevan: that's right, my bad. – larsmoa Jun 29 '11 at 18:06

score 1 · Answer 3 · answered Jun 29 '11 at 17:24

1

If you are only doing 8-bit lookups, it would be trivial to precalculate all your answers and then just store them in a 256 entry table. That way you would get constant time queries, and the memory storage would only be on the order of 256 entries.

answered Jun 29 '11 at 17:24

Mikola

9,176
2
34
41

1

Grr. Scooped me by 41 seconds. – Seth Robertson Jun 29 '11 at 17:25
1

Ah well, it was an easy question :) Have an upvote anyway. – Mikola Jun 29 '11 at 17:26

score 0 · Answer 4 · answered Jun 23 '11 at 17:32

0

If I understand, you're saying that if response has ONLY bits 1,3,5 set, then you want all codes in codebook that have flags 1,3,5 set and you don't care about bits 2,4,6,7,8.

If so, here's your pseudo code:

matchingCodes = new List<Code>
foreach(code in codebook)
    if((response & code) == response) matchingCodes.add(code);

answered Jun 23 '11 at 17:32

ean5533

8,884
3
40
64

Thanks for the reply... but as my initial post stated: "I have a standard linear search in place but of course it's pretty inefficient." I was hoping for some ideas on how to structure the data in a tree or other data structure for a faster search. – akevan Jun 29 '11 at 16:56

larsmoa · Answer 5 · 2011-06-29T18:45:28.377

I'm posting another answer because I got a new (and better suggestion):

Sort the codebook by value
For each code you're looking up, do first a binary lower bound search to find the first possible match (any value less than what you are looking up cannot be a match)
Search through the range from and including the first possible match until the last element to see if there are any matches.

The algorithm is still linear, but with an O(log N) lookup to cutoff (hopefully) most values. Lookup on small values would still be expensive, lookup on big values will be cheaper.

You could also maybe use a Bloom filter to do an initial search to cutoff most cases, and doing a linear search on the remaining. The filter may have false positives, so you would have to a linear search when the filter returns true. This datastructure requires you to have a good deal of independent hash functions (e.g. based on number of bits set, the product of all set bits, odd vs even, the number itself etc.). This could be a good optimization if you're expecting codes to be found only occasionally (if the filter returns false, the code is guaranteed to not be in the codebook). However, I suspect that this is more of theoretical interest than an actual optimization.

I don't think a bloom filter is useful. It would tell us whether that bit pattern was set, but not what values from that bit pattern were used. Since OP said that most bit patterns would be seen, I'm not seeing the benefit. — Seth Robertson, Jun 29 '11 at 18:49

searching bit-field templates (codebooks)

5 Answers5