1

Sorry for the bad title - I am trying to achieve the following: I have a data.table dt with two categorical variables "a" and "b". As you can see, a has 5 unique values and b has three. Now e.g. the combination of categorical variables ("a = 1" and "b = 3") is not in the data.

library(data.table) 
set.seed(1)
a <- sample(1:5, 10, replace = TRUE)
b <- sample(1:3, 10, replace = TRUE)
y <- rnorm(10)

dt <- data.table(a = a, b = b, y = y)
dt[order(a, b), .N, by = c("a", "b")]

#  a b N
#1: 1 1 2
#2: 1 2 1
#3: 2 2 1
#4: 2 3 1
#5: 3 1 1
#6: 3 2 1
#7: 3 3 1
#8: 4 1 1
#9: 5 2 1

If I simply sum "a" and "b", such groups as ("a = 1" and b = 3") will simply be ignored:

group_sum <- dt[, lapply(.SD, sum), by = c("a", "b")]
group_sum

#   a b          y
#1: 1 1 -0.7702614
#2: 4 1 -0.2894616
#3: 1 2 -0.2992151
#4: 2 2 -0.4115108
#5: 5 2  0.2522234
#6: 3 2 -0.8919211
#7: 2 3  0.4356833
#8: 3 1 -1.2375384
#9: 3 3 -0.2242679

Is there an internal way in data table to "keep" such missing groups and either assign a 0 or NA?

One way to achieve my goal would be to create a grid and merge in a second step:

grid <- unique(expand.grid(a = dt$a, b = dt$b)) # dim 
setDT(grid)

res <- merge(grid, group_sum, by = c("a", "b"), all.x = TRUE)
head(res)

#   a b          y
#1: 1 1 -0.7702614
#2: 1 2 -0.2992151
#3: 1 3         NA
#4: 2 1         NA
#5: 2 2 -0.4115108
#6: 2 3  0.4356833
A.Fischer
  • 596
  • 5
  • 11

2 Answers2

2

One way of going about this is to do a keyed cross-join with the CJ() function and then using .EACHI to note that y should be executed for every row in i.

library(data.table)

set.seed(1)
a <- sample(1:5, 10, replace = TRUE)
b <- sample(1:3, 10, replace = TRUE)
y <- rnorm(10)

dt <- data.table(a = a, b = b, y = y)
setkeyv(dt, c("a", "b"))

dt[CJ(a, b, unique = TRUE), lapply(.SD, sum), by = .EACHI]
#>     a b          y
#>  1: 1 1 -0.7702614
#>  2: 1 2 -0.2992151
#>  3: 1 3         NA
#>  4: 2 1         NA
#>  5: 2 2 -0.4115108
#>  6: 2 3  0.4356833
#>  7: 3 1 -1.2375384
#>  8: 3 2 -0.8919211
#>  9: 3 3 -0.2242679
#> 10: 4 1 -0.2894616
#> 11: 4 2         NA
#> 12: 4 3         NA
#> 13: 5 1         NA
#> 14: 5 2  0.2522234
#> 15: 5 3         NA

Created on 2020-10-03 by the reprex package (v0.3.0)

If you want to skip the key-setting step you could alternatively set the on argument:

dt <- data.table(a = a, b = b, y = y) # Set no key
dt[CJ(a, b, unique = TRUE), lapply(.SD, sum), by = .EACHI, on = c("a", "b")]
teunbrand
  • 33,645
  • 4
  • 37
  • 63
1

You can also use dplyr and tidyr with a complete() function:

library(dplyr)
library(tidyr)
dt %>% 
group_by(a,b) %>% 
complete(a,b) %>% 
summarize_all(sum) 
# A tibble: 15 x 3
# Groups:   a [5]
   a     b          y
   <fct> <fct>  <dbl>
 1 1     1      -6.93
 2 1     2      -2.69
 3 1     3      NA   
 4 2     1      NA   
 5 2     2      -3.70
 6 2     3       3.92
 7 3     1     -11.1 
 8 3     2      -8.03
 9 3     3      -2.02
10 4     1      -2.61
11 4     2      NA   
12 4     3      NA   
13 5     1      NA   
14 5     2       2.27
15 5     3      NA
StupidWolf
  • 45,075
  • 17
  • 40
  • 72