1

I'd like to aggregate a bunch of values that belong to a particular category into an HLL data structure so I can carry out intersections and unions later and count resulting cardinality of such computations.

I was able to get to the point where I can estimate cardinality for each group using the com.twitter.algebird.HyperLogLogAggregator

I need help using com.twitter.algebird.HyperLogLogMonoid to store as HLL and then later utilize to compute intersections/unions.

val lines_parsed = lines.map { line => parseBlueKaiLogEntry(line) }
# (uuid, [category id array])

val lines_parsed_flat = lines_parsed.flatMap { 
  case(uuid, category_list) => category_list.toList.map {
       category_id => (category_id, uuid) 
     }
}
# (category_id, uuid)

# Group by category
val lines_parsed_grped = lines_parsed_flat.groupBy { 
        case (cat_id, uuid) => cat_id 
      }

# Define HLL aggregator
val hll_uniq = HyperLogLogAggregator.sizeAggregator(bits=12).composePrepare[(String, String)]{case(cat_id, uuid) => uuid.toString.getBytes("UTF-8")}

# Aggregate using hll count
lines_parsed_grped.aggregate(hll_uniq).dump
# (category_id, count) - expected output

Now, I try to use HLL Monoid

# I now want to store as HLL and this is where I'm not sure what to do
# Create HLL Monoid
val hll = new HyperLogLogMonoid(bits = 12)

val lines_grped_hll = lines_parsed_grped.mapValues { case(cat_id:String, uuid:String ) =>  uuid}.values.map {v:String => hll.create(v.getBytes("UTF-8"))}

# Calling dump results in a lot more lines that I expect to see
lines_grped_hll.dump

What am I doing wring here ?

harshsinghal
  • 3,720
  • 8
  • 35
  • 32

1 Answers1

0

Use:

val result  =  hll.sum(lines_grped_hll) //or suitable method of hll for you

result.dump
FaigB
  • 2,271
  • 1
  • 13
  • 22