1

we are currently facing an interesting problem. We would like to estimate the cardinality of a set without the need to store every single item (typically bitmaps/bitsets are a nice approach). A very nice algorithm is the so called HyperLogLog randomized algorithm (see more here http://antirez.com/news/75).

The problem here is, that you can only merge sets as UNIONs, so basically it's a OR combination.

We actually want not only to combine sets with OR, but as well with AND. We even want to combine these operations.

Example: set1 AND (set2 OR set3) OR (set4 AND set5)

Each set may have a cardinality in the range of millions. Each value has a size of 128 bit.

Each set can be represented in any way e.g. "HLL, bloom filter, a plain list, or a combination of these". The algorithm must execute in the shortest possible amount of time using a feasible amount of space.

Any ideas?

Fritz
  • 872
  • 1
  • 8
  • 17
  • Do the sets need to be represented by only those structures or additional structures can be used? I mean, if you mix HLLs with MinHashes you can estimate the cardinality of set intersections pretty easily. – Juan Lopes May 27 '16 at 14:23

1 Answers1

2

This exact problem is the subject of https://pdfs.semanticscholar.org/5da8/bf81712187712aed159aed62e38fb012872e.pdf. Their recommendation is to use bloom filters.

The bloom filter for the union is the bitwise OR of the bloom filters. The bloom filter for intersection is the bitwise AND of the bloom filters. So you can easily generate the bloom filter of the operation that you want.

Their theorem 1 allows you to estimate the size of a set from how many bits are set in its bloom filter.

btilly
  • 43,296
  • 3
  • 59
  • 88
  • Nice! I will have a look into that... I am looking forward to see if this solution actually allows both operations (in combination) AND and OR. Thanks! – Fritz May 13 '16 at 08:37