R package development: tests pass in console, but fail via devtools::test()

Question

I am developing an R package that calls functions from the package rstan. As a MWE, my test file is currently set up like this, using code taken verbatim from rstan's example:

library(testthat)
library(rstan)

# stan's own example
stancode <- 'data {real y_mean;} parameters {real y;} model {y ~ normal(y_mean,1);}'
mod <- stan_model(model_code = stancode, verbose = TRUE)
fit <- sampling(mod, data = list(y_mean = 0))
# I added this line, and it's the culprit
summary(fit)$summary

When I run this code in the console or via the "Run Tests" button in RStudio, no errors are thrown. However, when I run devtools::test(), I get:

Error (test_moments.R:11:1): (code run outside of `test_that()`)
Error in `summary(fit)$summary`: $ operator is invalid for atomic vectors

and this error is definitely not occurring upstream of that final line of code, because removing the final line allows devtools::test() to run without error. I am running up-to-date packages devtools and rstan.

I'm not sure why you were downvoted, but your question _could_ be improved by including code that creates a minimal R package containing the failing test. — Mikael Jagan, Nov 11 '21 at 15:48

Mikael Jagan · Accepted Answer · 2021-11-12T01:10:15.300

It seems that devtools::test evaluates the test code in a setting where S4 dispatch does not work in the usual way, at least for packages that you load explicitly in the test file (in this case rstan). As a result, summary dispatches to summary.default instead of the S4 method implemented in rstan for class "stanfit".

The behaviour that you're seeing might relate to this issue on the testthat repo, which seems unresolved.

Here is a minimal example that tries to illuminate what is happening, showing one possible (admittedly inconvenient) work-around.

pkgname <- "foo"
usethis::create_package(pkgname, rstudio = FALSE, open = FALSE)
setwd(pkgname)

usethis::use_testthat()
path_to_test <- file.path("tests", "testthat", "test-summary.R")
text <- "test_that('summary', {
  library('rstan')
  stancode <- 'data {real y_mean;} parameters {real y;} model {y ~ normal(y_mean,1);}'
  mod <- stan_model(model_code = stancode, verbose = TRUE)
  fit <- sampling(mod, data = list(y_mean = 0))

  expect_identical(class(fit), structure('stanfit', package = 'rstan'))
  expect_true(existsMethod('summary', 'stanfit'))

  x <- summary(fit)
  expect_error(x$summary)
  expect_identical(x, summary.default(fit))
  print(x)

  f <- selectMethod('summary', 'stanfit')
  y <- f(fit)
  str(y)
})
"
cat(text, file = path_to_test)

devtools::test(".") # all tests pass

If your package actually imports rstan (in the NAMESPACE sense, not in the DESCRIPTION sense), then S4 dispatch seems to work fine, presumably because devtools loads your package and its dependencies in a "proper" way before running any tests.

cat("import(rstan)\n", file = "NAMESPACE")
newtext <- "test_that('summary', {
  stancode <- 'data {real y_mean;} parameters {real y;} model {y ~ normal(y_mean,1);}'
  mod <- stan_model(model_code = stancode, verbose = TRUE)
  fit <- sampling(mod, data = list(y_mean = 0))

  x <- summary(fit)
  f <- selectMethod('summary', 'stanfit')
  y <- f(fit)
  expect_identical(x, y)
})
"
cat(newtext, file = path_to_test)

## You must restart your R session here. The current session 
## is contaminated by the previous call to 'devtools::test',
## which loads packages without cleaning up after itself...

devtools::test(".") # all tests pass

If your test is failing and your package imports rstan, then something else may be going on, but it is difficult to diagnose without a minimal version of your package.

Disclaimer: Going out of your way to import rstan to get around a relatively obscure devtools issue should be considered more of a hack than a fix, and documented accordingly...

Perfect! Using import(rstan) in NAMESPACE instead of importFrom(...) solved it. Much appreciated. — half-pass, Nov 11 '21 at 23:21
I actually assumed that you didn't have `import` _or_ `importFrom`, and figured that having either in your `NAMESPACE` would be sufficient. It's interesting that only `import` works. In any case, a couple of remarks: (1) If you are documenting your package with `roxygen2`, then you shouldn't edit `NAMESPACE` by hand. You just need `#' @import rstan` somewhere in your package. (2) You may want to document why you are using `import` over `importFrom`, because my answer relies on (to my knowledge) an undocumented behaviour of `devtools`... — Mikael Jagan, Nov 12 '21 at 00:13

R package development: tests pass in console, but fail via devtools::test()

1 Answers1