How to manipulate NULL elements in a nested list?

Question

I have a nested list containing NULL elements, and I'd like to replace those with something else. For example:

l <- list(
  NULL,
  1,
  list(
    2,
    NULL,
    list(
      3,
      NULL
    )
  )
)

I want to replace the NULL elements with NA. The natural way to do this is to recursively loop over the list using rapply. I tried:

rapply(l, function(x) NA, classes = "NULL", how = "replace")
rapply(l, function(x) if(is.null(x)) NA else x, how = "replace")

Unfortunately, neither of these methods work, since rapply apparently ignores NULL elements.

How can I manipulate the NULL elements in a nested list?

I finally gave up on rapply and use [something which I can predict how it will work](https://github.com/raredd/rawr/blob/master/R/utils.R#L1372-L1381) so `rapply2(l, function(x) if (is.null(x)) NA else x)` would work — rawr, Aug 15 '16 at 06:22
Here is an interesting [post](http://stackoverflow.com/questions/7170264/why-do-rapply-and-lapply-handle-null-differently) that talks about this some but doesn't quite answer your question of how to get around the issue. — steveb, Aug 15 '16 at 06:25
at steveb; it is on point. I'm sorry @Owen doesn't seem to be spending much time here anymore. His posts were always very insightful. — IRTFM, Aug 15 '16 at 06:29
Basically everyone thought to do the same thing and rewrite their own recursive lapply. A little overburdensome IMHO; perhaps this should exist in base R. — shayaa, Aug 15 '16 at 06:50
@rawr I wonder if you have changed the behavior of `rapply2()` since you wrote this comment. — jazzurro, Sep 02 '19 at 13:06
@jazzurro yes, it appears I have, nice catch. In the current version if you remove the `if (is.null(l[[ii]])) next` line it should work. the original is [here](https://github.com/raredd/rawr/commit/b077d91fa107cb9ce651c0fca296dff8447608dc#diff-cbb7235afee10c3646161358a7655b0a) which doesn't include that line. I should add a logical to skip null elements or not — rawr, Sep 10 '19 at 02:52
@rawr Thanks for that. I look forward to using an updated version soon! — jazzurro, Sep 10 '19 at 03:09

Richie Cotton · Accepted Answer · 2016-08-15T06:37:17.150

I'm going to go with "use a version of rapply doesn't doesn't have weird behaviour with NULL". This is the simplest implementation I can think of:

simple_rapply <- function(x, fn)
{
  if(is.list(x))
  {
    lapply(x, simple_rapply, fn)
  } else
  {
    fn(x)
  }
}

(rawr::rapply2, as mentioned in the comments by @rawr is a more sophisticated attempt.)

Now I can do the replacement using

simple_rapply(l, function(x) if(is.null(x)) NA else x)

score 11 · Answer 2 · answered Aug 15 '16 at 06:46

This is what William Dunlap suggested in 2010 when this question was asked on Rhelp:

replaceInList <- function (x, FUN, ...) 
  {
      if (is.list(x)) {
          for (i in seq_along(x)) {
              x[i] <- list(replaceInList(x[[i]], FUN, ...))
          }
          x
      }
      else FUN(x, ...)
  }
 replaceInList(l, function(x)if(is.null(x))NA else x)

Jonathan Carroll · Answer 3 · 2016-08-15T06:37:49.397

6

This is a hack, but as far as hacks go, I think I'm somewhat happy with it.

lna <- eval(parse(text = gsub("NULL", "NA", deparse(l))))

str(lna)
#> List of 3
#> $ : logi NA
#> $ : num 1
#> $ :List of 3
#> ..$ : num 2
#> ..$ : logi NA
#> ..$ :List of 2
#> .. ..$ : num 3
#> .. ..$ : logi NA

Update:

If for some reason you needed "NULL" as a character entry in the list (corner case, much?) you can still use the above hack since it replaces the contents of the string, not the quotes, thus it just requires another step

l2 <- list(
  NULL,
  1,
  list(
    2,
    "NULL",
    list(
      3,
      NULL
    )
  )
)

lna2   <- eval(parse(text = gsub("NULL", "NA", deparse(l2))))
lna2_2 <- eval(parse(text = gsub('\\"NA\\"', '\"NULL\"', deparse(lna2))))

str(lna2_2)
#> List of 3
#> $ : logi NA
#> $ : num 1
#> $ :List of 3
#> ..$ : num 2
#> ..$ : chr "NULL"
#> ..$ :List of 2
#> .. ..$ : num 3
#> .. ..$ : logi NA

edited Aug 15 '16 at 06:37

answered Aug 15 '16 at 06:12

Jonathan Carroll

3,897
14
34

That's a very nice hack, but I worry about the dark magic in `eval`. So the example is fine, but a string named `"NULL"` will confuse this. – Richie Cotton Aug 15 '16 at 06:23
Absolutely. Is that of concern to your use-case? Is that terrible practice to begin with? – Jonathan Carroll Aug 15 '16 at 06:25
3

It's kind of amusing to see this effort to convert R to a macro-processor. Not "bad", just amusing. – IRTFM Aug 15 '16 at 06:26
Trying out what would happen with the string "NULL", the quotes are untouched, so really you just need a second step... Editing. – Jonathan Carroll Aug 15 '16 at 06:36
@JonathanCarroll Another evil corner case: The second version breaks if one of the names of the elements is `"NA"`. (That name gets converted to `"NULL"`. – Richie Cotton Aug 15 '16 at 06:58
@42- I think it is bad. And (depending on the content of the list) potentially extremely inefficient. Just imagine deparsing/parsing a list containing some large objects. – Roland Aug 15 '16 at 06:58
@Roland "bad" is maybe a little harsh. This is a decent solution for the use case where you have a specific list that you want to process. For a very large list, or a general list where you don't know the contents, it isn't suitable. – Richie Cotton Aug 15 '16 at 07:03
I'm not doubting for a moment that this is a poorer solution than using a better version of `rapply`. My answer is aimed at the question as posed (and works just fine for that), not numerous "but if"s that follow (especially ones that involve `"NA"` and `"NULL"` as plausible scenarios). – Jonathan Carroll Aug 15 '16 at 07:03

shayaa · Answer 4 · 2016-08-15T07:09:38.243

5

I wrapped the replacement inside the sapply, which makes it more readable/understandable to me, albeit less general.

 replace_null <- function(x) {
  lapply(x, function(x) {
    if (is.list(x)){
      replace_null(x)
      } else{
        if(is.null(x)) NA else(x)
      } 
    })
}

replace_null(l)

edited Aug 15 '16 at 07:09

answered Aug 15 '16 at 06:43

shayaa

2,787
13
19

Instead of `sapply` with `simplify = FALSE` just use `lapply`. You should also avoid `ifelse`. – Roland Aug 15 '16 at 07:00
@shayaa Using `ifelse` rather than `if` and `else` means that you'll have problems for non-vector inputs. Try including a data.frame or a formula as one element of the list. – Richie Cotton Aug 15 '16 at 07:05
I get it; and it provides no vectorization benefits anyways, since it's not as if one of these list elements can be `c(1,NULL,1)` because the vector will just remove that element. I suppose that writing `if(is.null(x)) NA else(x)` just seemed clunkier to me. I will edit nonetheless – shayaa Aug 15 '16 at 07:09

Joris C. · Answer 5 · 2020-10-12T18:05:17.597

This can also be done with rrapply() in the rrapply-package. Below are a few different ways we could replace the NULL elements in a nested list by NA values:

library(rrapply)

l <- list(
    NULL,
    1,
    list(
        2,
        NULL,
        list(
            3,
            NULL
        )
    )
)

## replace NULL by NA using only f
rrapply(l, f = function(x) if(is.null(x)) NA else x, how = "replace")
#> [[1]]
#> [1] NA
#> 
#> [[2]]
#> [1] 1
#> 
#> [[3]]
#> [[3]][[1]]
#> [1] 2
#> 
#> [[3]][[2]]
#> [1] NA
#> 
#> [[3]][[3]]
#> [[3]][[3]][[1]]
#> [1] 3
#> 
#> [[3]][[3]][[2]]
#> [1] NA

## replace NULL by NA using condition argument
rrapply(l, condition = is.null, f = function(x) NA, how = "replace")
#> [[1]]
#> [1] NA
#> 
#> [[2]]
#> [1] 1
#> 
#> [[3]]
#> [[3]][[1]]
#> [1] 2
#> 
#> [[3]][[2]]
#> [1] NA
#> 
#> [[3]][[3]]
#> [[3]][[3]][[1]]
#> [1] 3
#> 
#> [[3]][[3]][[2]]
#> [1] NA

## replace NULL by NA using condition and deflt arguments 
rrapply(l, condition = Negate(is.null), deflt = NA, how = "list")
#> [[1]]
#> [1] NA
#> 
#> [[2]]
#> [1] 1
#> 
#> [[3]]
#> [[3]][[1]]
#> [1] 2
#> 
#> [[3]][[2]]
#> [1] NA
#> 
#> [[3]][[3]]
#> [[3]][[3]][[1]]
#> [1] 3
#> 
#> [[3]][[3]][[2]]
#> [1] NA

We can also prune the NULL elements from the list altogether by setting how = "prune":

## keep only non-NULL elements
rrapply(l, condition = Negate(is.null), how = "prune")
#> [[1]]
#> [1] 1
#> 
#> [[2]]
#> [[2]][[1]]
#> [1] 2
#> 
#> [[2]][[2]]
#> [[2]][[2]][[1]]
#> [1] 3

How to manipulate NULL elements in a nested list?

5 Answers5

Linked