0

If I create a HyperLogLog per day to count unique visitors, and then the 1st of January I merge the last 365 ones will I get the same value than if I keep a single HyperLogLog for the whole 365 days?

I guess not. But how different would those values be?

vtscop
  • 31
  • 11

1 Answers1

3

Short answer: Yes, both solutions have the same accuracy.

The basis of the HyperLogLog algorithm is the observation that the cardinality of a multiset of uniformly distributed random numbers can be estimated by calculating the maximum number of leading zeros in the binary representation of each number in the set. If the maximum number of leading zeros observed is n, an estimate for the number of distinct elements in the set is 2**n.

Referred from wiki

No matter it's a single hyperloglog or 365 hyperloglogs, the numbers of leading zeros are the same. So they have the same accuracy, and that's why hyperloglog can be merged.

for_stack
  • 21,012
  • 4
  • 35
  • 48
  • I'm not sure if this is correct. From a statistical standpoint, the merge operation introduces some bias to the estimator. – Lior Kogan Dec 18 '22 at 09:15
  • 1
    @LiorKogan after all, the state of the merged HyperLogLog will be identical to that of the single HyperLogLog. Therefore, the estimate will be identical as well. The only exception is, if you use the martingale estimator (see https://arxiv.org/abs/1306.3284 or https://dl.acm.org/doi/abs/10.1145/2623330.2623669) for non-distributed data. In this case you will get a more accurate estimate, if there are no merge operations involved. – otmar Dec 19 '22 at 09:07