R: aggregate values on a tree

Question

This question is similar to this, but it's got a C# answer, and I need a R answer.

I have some 50 files of about 650 rows with a format and data very similar to this toy data:

dput(y)
structure(list(level1 = c(4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L), level2 = c(NA, 41L, 41L, 41L, 41L, 41L, 41L, 41L, 
42L, 42L, 42L, 42L), level3 = c(NA, NA, 4120L, 4120L, 4120L, 
4120L, 4120L, 4120L, NA, 4210L, 4210L, 4210L), level4 = c(NA, 
NA, NA, 412030L, 412030L, 412050L, 412050L, 412050L, NA, NA, 
421005L, 421005L), pid = c(NA, NA, NA, NA, 123456L, NA, 789012L, 
345678L, NA, NA, NA, 901234L), description = c("income", "op.income", 
"manuf.industries", "manuf 1", "client 1", "manuf 2", "client 2", 
"client 3", "non-op.income", "financial", "interest", "bank 1"
), value = c(NA, NA, NA, NA, 15000L, NA, 272860L, 1150000L, NA, 
NA, NA, 378L)), .Names = c("level1", "level2", "level3", "level4", 
"pid", "description", "value"), class = c("data.table", "data.frame"
), row.names = c(NA, -12L), .internal.selfref = <pointer: 0x00000000001a0788>)

Each of the rows that have a value on value are a "leaf" o a tree, with branches identified in columns level1 to 4. I want to summarize the leafs by brach and put the corresponding values in the value column.

My expected output looks like this:

dput(res)
structure(list(level1 = c(4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L), level2 = c(NA, 41L, 41L, 41L, 41L, 41L, 41L, 41L, 
42L, 42L, 42L, 42L), level3 = c(NA, NA, 4120L, 4120L, 4120L, 
4120L, 4120L, 4120L, NA, 4210L, 4210L, 4210L), level4 = c(NA, 
NA, NA, 412030L, 412030L, 412050L, 412050L, 412050L, NA, NA, 
421005L, 421005L), pid = c(NA, NA, NA, NA, 123456L, NA, 789012L, 
345678L, NA, NA, NA, 901234L), description = c("income", "op.income", 
"manuf.industries", "manuf 1", "client 1", "manuf 2", "client 2", 
"client 3", "non-op.income", "financial", "interest", "bank 1"
), value = c(1438238L, 1437860L, 1437860L, 15000L, 15000L, 1422860L, 
272860L, 1150000L, 378L, 378L, 378L, 378L)), .Names = c("level1", 
"level2", "level3", "level4", "pid", "description", "value"), class = c("data.table", 
"data.frame"), row.names = c(NA, -12L), .internal.selfref = <pointer: 0x00000000001a0788>)

I know this can be done with a for-loop, but I wanted to know if there is any faster, simpler alternative (I prefer data.table or base-solutions, but any other package works ok too). What I've tried so far:

z4<-y[!is.na(pid),sum(value),by=level4]
setkey(y,"level4");setkey(z4,"level4")
y[z4,][is.na(pid)]

This shows me the desired values in V1, so I wanted to see if I could assign them to value:

y[z4,][is.na(pid),value:=i.V1]
Error in eval(expr, envir, enclos) : object 'i.V1' not found

I think this could be caused because the call i.V1 is in the chained [ and not in the initial y[z4 call. But if I only subset on z4, how can I know which of the several matching level4 rows I should assign (that's why I'm thinking of using is.na(pid), because y[z4,value:=i.V1] produces the wrong result, as it updates all values that match level4).

As you can see, I'm badly stuck at this problem, and with "my method" I still would have 3 more levels to go.

Is there any easier way to do this?

Answer 1

Because the computations at each level require those from the previous level, I think a loop or recursion is required. Here is a recursive function to get the values using base R. You could surely do something similar with data.table, which would probably be much more efficient.

## Use y as data.frame
y <- as.data.frame(y)

## Recursive function to get values
f <- function(data, lvl=NULL) {
    if (is.null(lvl)) lvl <- 1                                                 # initialize level
    if (lvl == 5) return (data)                                                # we are done
    cname <- paste0("level", lvl)                                              # name of current level
    nname <- ifelse (lvl == 4, "pid", paste0("level", lvl+1))                  # name of next level
    agg <- aggregate(as.formula(paste("value~", cname)), data=data, sum)       # aggregate data
    inds <- (ms <- match(data[,cname], agg[,cname], F)) & is.na(data[,nname])  # find index of leaves to fill
    data$value[inds] <- agg$value[ms[inds]]                                    # add new values
    f(data, lvl+1)                                                             # recurse
}

f(data=y)
#    level1 level2 level3 level4    pid      description   value
# 1       4     NA     NA     NA     NA           income 1438238
# 2       4     41     NA     NA     NA        op.income 1437860
# 3       4     41   4120     NA     NA manuf.industries 1437860
# 4       4     41   4120 412030     NA          manuf 1   15000
# 5       4     41   4120 412030 123456         client 1   15000
# 6       4     41   4120 412050     NA          manuf 2 1422860
# 7       4     41   4120 412050 789012         client 2  272860
# 8       4     41   4120 412050 345678         client 3 1150000
# 9       4     42     NA     NA     NA    non-op.income     378
# 10      4     42   4210     NA     NA        financial     378
# 11      4     42   4210 421005     NA         interest     378
# 12      4     42   4210 421005 901234           bank 1     378

I think the aggregation step could be made more efficient by only aggregating a subset of the data if need be. Honestly, this was fun, but a loop is probably the way to go.