6

In my understanding, anything put inside test_that() should be compartmentalized, meaning that if I load a package in test_that(), its functions and methods shouldn't be available in other test_that() calls.

In the example below, there are 3 (empty) tests:

  • In the first one, we can see that the method as.data.frame.lm is not available in the namespace.
  • In the second one, I load the package parameters, which provides the method as.data.frame.lm. In my understanding, this method should only be available in this test_that() call.
  • In the third one, we can see that the method is available.

Edit: the package parameters no longer exports this method so I modified the example below but the rationale stays the same.

library(testthat)

test_that("foo 1", {
  print("as.matrix.get_predicted" %in% methods(as.matrix))
})
#> [1] FALSE
#> ── Skip (???): foo 1 ───────────────────────────────────────────────────────────
#> Reason: empty test

test_that("foo 2", {
  invisible(insight::get_parameters)
})
#> ── Skip (???): foo 2 ───────────────────────────────────────────────────────────
#> Reason: empty test

test_that("foo 3", {
  print("as.matrix.get_predicted" %in% methods(as.matrix))
})
#> [1] TRUE
#> ── Skip (???): foo 3 ───────────────────────────────────────────────────────────
#> Reason: empty test

Why is that? Are there some workarounds?

Edit: I'm looking for a solution specific to testthat, not another testing framework.

bretauv
  • 7,756
  • 2
  • 20
  • 57
  • 1
    In R, package loading is not scoped to functions. Running `library()` has global side effects, You'd have to explicitly unload the packages/namespaces yourself. Probably best dealt with via a test fixture: https://testthat.r-lib.org/articles/test-fixtures.html – MrFlick Mar 23 '23 at 13:38
  • Thanks, but it seems weird to me that there's no built-in functions to make sure of this. To me loading other packages in tests is quite standard but it clearly has side effects. I also saw that the package `withr` has [`with_package()`](https://withr.r-lib.org/reference/with_package.html) and `local_package()` but they don't unload the package namespace so I don't really understand their purpose – bretauv Mar 23 '23 at 14:06
  • There is `detach("package:parameters", unload = TRUE)`, but that won't unregister S3 methods or unload the dependencies of **parameters** that are loaded automatically by `library(parameters)`. The correct approach is to run the relevant tests in a new R process, perhaps by having more than one `*.R` file under `tests/` (what people who spurn **testthat** do). – Mikael Jagan Mar 23 '23 at 23:46
  • Even if I put the test that loads `parameters` in another test file (and if this test file is ran before the others), the S3 method will still be available in other test files so that doesn't work. – bretauv Mar 24 '23 at 07:09

2 Answers2

6

Too long for a comment, but reproducible. The test files are sourced in collation order, each in a new R process, hence the library call in testB.R does not cause the test in testC.R to fail.

pkgname <- "testpackage"
testdir <- file.path(pkgname, "tests")

.add <- function(a, b) a + b
package.skeleton(pkgname, list = ".add")
dir.create(testdir, recursive = TRUE)
writeLines("stopifnot(!any(.S3methods(as.data.frame) == \"as.data.frame.lm\"))",
           file.path(testdir, "testA.R"))
writeLines("library(parameters); stopifnot(any(.S3methods(as.data.frame) == \"as.data.frame.lm\"))",
           file.path(testdir, "testB.R"))
writeLines("stopifnot(!any(.S3methods(as.data.frame) == \"as.data.frame.lm\"))",
           file.path(testdir, "testC.R"))

tools::Rcmd(c("check", pkgname))

* checking tests ...
  Running ‘testA.R’
  Running ‘testB.R’
  Running ‘testC.R’
 OK

This is the "vanilla" approach to compartmentalizing tests that many people still prefer over testthat and other frameworks, at least partly because it heeds the warning in ?detach:

The most reliable way to completely detach a package is to restart R.

and therefore is significantly easier for experts to debug and reason about, but that is perhaps a controversial opinion these days ...

Mikael Jagan
  • 9,012
  • 2
  • 17
  • 48
  • Thank you for your answer but I'm not going to replace `testthat` by another approach given the amount of tests. Surely I'm not the only one running into this issue so I guess there must be some `testthat`-specific solution – bretauv Mar 27 '23 at 17:01
  • 2
    It is a documented limitation of the R language: there is no reliable way to reverse the global effects of loading a package. If **testthat** had a magic answer, then it would be mentioned in `vignette("test-fixtures")`, but there simply isn't one, as stated in `help("detach")`. IMO, that vignette needs to be much more transparent about the limitations of **testthat** w.r.t. the global effects of `loadNamespace` calls. It is really a glaring omission. – Mikael Jagan Mar 27 '23 at 18:35
0

As a complement to @MikaelJagan's answer, there is a section in "R Packages" that addresses this problem:

It’s fair to say that library(somePkg) in the tests should be about as rare as taking a dependency via Depends, i.e. there is almost always a better alternative.

Unnecessary calls to library(somePkg) in test files have a real downside, because they actually change the R landscape. library() alters the search path. This means the circumstances under which you are testing may not necessarily reflect the circumstances under which your package will be used. This makes it easier to create subtle test bugs, which you will have to unravel in the future.

bretauv
  • 7,756
  • 2
  • 20
  • 57