Fast top N by count by group in data.table

Question

I'd like to know the preferred way to frank subgroups on the count of their appearances by group.

For example, I have customers who belong to segments and who have postal codes. I would like to know the most common 3 postal codes for each segment.

library(data.table)
set.seed(123)
n <- 1e6
df <- data.table( cust_id = 1:n,
                  cust_segment = sample(LETTERS, size=n, replace=T),
                  cust_postal = sample(as.character(5e4:7e4),size=n, replace=T)
                 )

This chain (inside the dcast() below) produces the desired output but requires two passes, the first to count by group-subgroup and the second to rank the counts by group.

dcast(
  df[,.(.N),
     by = .(cust_segment, cust_postal)
     ][,.(cust_postal,
          postal_rank = frankv(x=N, order=-1, ties.method = 'first')
     ), keyby=cust_segment
     ][postal_rank<=3],
  cust_segment ~ paste0('postcode_rank_',postal_rank), value.var = 'cust_postal' 
)
# desired output:
# cust_segment postcode_rank_1 postcode_rank_2 postcode_rank_3
#            A           51274           64588           59212
#            B           63590           69477           50380
#            C           60619           66249           53494 ...etc...

Is that the best there is, or is there a single-pass approach?

Seems fine, though I guess you should `:=` frankv instead of making a new table. Using one call to `forder` instead of many to `frankv` might also be faster: `df[, .N, keyby=.(cust_segment, cust_postal)][order(-N), r := rowid(cust_segment)][r <= 3, dcast(.SD, cust_segment ~ r, value.var="cust_postal")]` (Using keyby= since Matt mentioned that it's faster than by=) — Frank, Jul 06 '18 at 20:29

score 1 · Accepted Answer · answered Jul 07 '18 at 07:49

Taking the answer from Frank out of the comments:

Using forder instead of frankv and using keyby as this is faster than just using by

df[, .N, 
   keyby = .(cust_segment, cust_postal)
   ][order(-N), r := rowid(cust_segment)
     ][r <= 3, dcast(.SD, cust_segment ~ r, value.var ="cust_postal")]

    cust_segment     1     2     3
 1:            A 51274 53440 55754
 2:            B 63590 69477 50380
 3:            C 60619 66249 52122
 4:            D 68107 50824 59305
 5:            E 51832 65249 52366
 6:            F 51401 55410 65046

microbenchmark time:

library(microbenchmark)

microbenchmark(C8H10N4O2 = dcast(
                                df[,.(.N),
                                   by = .(cust_segment, cust_postal)
                                   ][,.(cust_postal,
                                        postal_rank = frankv(x=N, order=-1, ties.method = 'first')
                                   ), keyby=cust_segment
                                   ][postal_rank<=3],
                                cust_segment ~ paste0('postcode_rank_',postal_rank), value.var = 'cust_postal' 
                              ),
              frank = df[, .N, 
                         keyby = .(cust_segment, cust_postal)
                         ][order(-N), r := rowid(cust_segment)
                           ][r <= 3, dcast(.SD, cust_segment ~ r, value.var ="cust_postal")])
Unit: milliseconds
     expr      min       lq     mean   median       uq      max neval
C8H10N4O2 136.3318 140.8096 156.2095 145.6099 170.4862 205.8457   100
    frank 102.2789 110.0140 118.2148 112.6940 119.2105 192.2464   100

Frank's answer is about 25% faster.

Fast top N by count by group in data.table

1 Answers1