Error when performing an NA replacement in R 4.0

Question

With R 3.6 I can perform the following NA replacement

> d <- zoo(data.frame(a = NA, b = 1), Sys.Date())
> d[is.na(d)] <- 1
> d
           a b
2021-03-03 1 1

With R 4.0 I get the following error:

> d <- zoo(data.frame(a = NA, b = 1), Sys.Date())
> d[is.na(d)] <- 1
Error in as.Date.default(e) : 
  do not know how to convert 'e' to class “Date”

Has some default behavior changed in R 4.0?

R 3.6 session info:

Microsoft Windows [Version 10.0.19041.804]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\>R --no-site-file

R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: i386-w64-mingw32/i386 (32-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(zoo)

Attaching package: 'zoo'

The following objects are masked from 'package:base':

    as.Date, as.Date.numeric

Warning message:
package 'zoo' was built under R version 4.0.4
> d <- zoo(data.frame(a = NA, b = 1), Sys.Date())
> d[is.na(d)] <- 1
> d
           a b
2021-03-03 1 1

R 4.0 session info:

Microsoft Windows [Version 10.0.19041.804]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\>R --no-site-file

R version 4.0.4 (2021-02-15) -- "Lost Library Book"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: i386-w64-mingw32/i386 (32-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(zoo)

Attaching package: 'zoo'

The following objects are masked from 'package:base':

    as.Date, as.Date.numeric

> d <- zoo(data.frame(a = NA, b = 1), Sys.Date())
> d[is.na(d)] <- 1
Error in as.Date.default(e) :
  do not know how to convert 'e' to class "Date"

Session Info (3.6):

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 10 x64 (build 19041)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] zoo_1.8-8

loaded via a namespace (and not attached):
[1] compiler_3.6.1  grid_3.6.1      lattice_0.20-38

Session Info (4.0):

> sessionInfo()
R version 4.0.4 (2021-02-15)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 10 x64 (build 19041)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] zoo_1.8-8

loaded via a namespace (and not attached):
[1] compiler_4.0.4  tools_4.0.4     grid_4.0.4      lattice_0.20-41

I would guess this has less to do with the version of R and more the version of the `zoo` package you have installed. If you run `sessionInfo()` after loading zoo, what version are you using in the different R consoles? — MrFlick, Mar 04 '21 at 04:51
I've found an online compiler *R3.4.4, zoo 1.8.1* and can replicate the old behavior, might indeed be a bug or changed routine. Try `d <- na.fill(d, 1)` to get what you want. — jay.sf, Mar 04 '21 at 05:03
@jay.sf na.fill() does work. Would the bug be in the zoo package? My code base uses is.na() in many places. I can obviously update the code, but this will be a significant issue for many people, I would imagine... — billelev, Mar 04 '21 at 05:07
@jay.sf Where is the online compiler? Is it that the output of `is.na(d)` is different? — MrFlick, Mar 04 '21 at 05:07
It's [here](https://rextester.com/l/r_online_compiler) @MrFlick — jay.sf, Mar 04 '21 at 05:09
The only difference I can see is that `is.na(d)` is class "matrix" in R 3.6 and classes "matrix" "array" in R 4.0. Perhaps that affects sub-setting in zoo? — neilfws, Mar 04 '21 at 21:03
@neilfws, yes. Achim looked at it and determined that that change in R is the cause of this. — G. Grothendieck, Mar 05 '21 at 14:32

score 3 · Accepted Answer · answered Mar 06 '21 at 02:06

Thanks for raising this issue, it was a bug in the zoo package. In the [.zoo and [<-.zoo methods we checked whether the index i was a matrix via

if (all(class(i) == "matrix")) ...

This worked correctly in R 3.x.y because matrix objects just had the class "matrix". However, in R 4.0.0 matrix objects started additionally inheriting from "array". See: https://developer.R-project.org/Blog/public/2019/11/09/when-you-think-class.-think-again/.

In the zoo development version on R-Forge (https://R-Forge.R-project.org/R/?group_id=18) I have fixed the issue now by replacing the above code with

if (inherits(i, "matrix")) ...

So you can already install zoo 1.8-9 from R-Forge and your code will work again as intended. Alternatively, you can wait for that version to arrive on CRAN which will hopefully come out in the next days after reverse dependency checks. In the mean time you can work around the issue by using

coredata(d)[is.na(d)] <- 1

This downloads and installs the development version of zoo: `install.packages("zoo", repos="http://R-Forge.R-project.org")` — G. Grothendieck, Mar 07 '21 at 13:58
The fixed version 1.8-9 has also been released on CRAN now: https://CRAN.R-project.org/package=zoo — Achim Zeileis, Mar 09 '21 at 16:45

score 0 · Answer 2 · answered Mar 05 '21 at 06:52

I'm having an issue with this as well!! here's more odd behavior/breadcrumbs as to what might be happening. Still not sure WHY but seems to be an indexing issue w/ zoo rather than is.na() specifically. Logical indexing works if the structure of the logical has the same rownames/indices as the zoo obj:

Printing out d[is.na(d)] (without assignment) results in an empty zoo object, suggesting the issue is w/ indexing
wrapping d in coredata() works

d <- zoo(data.frame(a = NA, b = 1), Sys.Date())
coredata(d)[is.na(d)] <- 1
d

             a b
> 2021-03-05 1 1

The logical returned by is.na() will work if it is transformed to have the same rownames/indices as the zoo obj.

d <- zoo(data.frame(a = NA, b = 1), Sys.Date())
changes <- is.na(d) #storing logical in a variable

> d
            a b
2021-03-05 NA 1

> changes #d[changes] won't work, so change rownames 
        a     b
[1,] TRUE FALSE

> changes <- as.zoo(changes, index(d))
> changes
              a     b
2021-03-05 TRUE FALSE

> d[as.logical(changes)] #changing zoo back to a logical, returns something 
            a b
2021-03-05 NA 1

> d[as.logical(changes)] <- 1
            a b
2021-03-05 NA 1

Now for the breadcrumbs...does anybody know what changes were made to R's date class in version 4? Zoo suggest it made some changes to merge.zoo "explicitly to work around the new behavior of c.Date() in R >= 4.1.0."

https://cran.r-project.org/web/packages/zoo/NEWS (see bullet point #2 at the top)

I've searched and searched and see no mention of those changes...

I'm guessing there were some changes to the zoo class to more strictly enforce date indexing...unsure...also can't quite seem to figure out to post an issue w/ zoo

more breadcrumbs.... apparently this functionality was working for zoo objects back in April of 2020 according to this thread https://github.com/joshuaulrich/xts/issues/331

R 4.0.1 came out later that month and newest zoo package, 1.8-8, was released 5/2020 so maybe running version 1.8-7 of this package could determine if it's a change w/ R or a change w/ zoo that's causing different behavior

score 0 · Answer 3 · answered Mar 05 '21 at 16:55

@neilfws you are correct that the issue is due to the change in class response in R 4.0.

For now, the best option is to use either:

d <- na.fill(d, 1)

or

coredata(d)[is.na(d)] <- 1

The old use case of d[is.na(d)] <- 1 will require an update to the zoo package

Error when performing an NA replacement in R 4.0

3 Answers3