Handling exceptions using tryCatch inside future.apply::future_lapply

Question

I want to reattempt failing readLines fetches using tryCatch. This works as expected, as long as I don't wrap it inside a future.apply::future_lapply call for processing a list or vector.

The problem can be reproduced using this code:

read_lines_retrying <- function(url, attempts = 5, throttle = 5) {
    result <- NA
    while (is.na(result) && 0 < attempts) {
        attempts <- attempts - 1
        result <- tryCatch(
            {
                readLines(url)
            },
            error = function(cond) {
                message("caught error:")
                message(cond)
                message("")
                Sys.sleep(throttle)
                return(NA)
            }
        )
    }
    if (is.na(result)) {
        stop(paste("could not get URL ", url))
    }
    return(result)
}

urls <- c("http://nonexistant.nonexistant")

future.apply::future_lapply(urls, read_lines_retrying)

Of course, the code is meant to retry on transient readLines failures, while the example URL will always fail, but this way problem can be most easily seen. When using lapply instead of future.apply::future_lapply, it takes at least 5 seconds to complete because it waits 5 seconds after each of the 5 attempts. This in not the case with future.apply::future_lapply, demonstrating that the exception handling doesn't work.

What am I doing wrong, and how can I get tryCatch to work inside future.apply::future_lapply?

As explained in my answer below, it is incorrect to use `message(cond)` here. The proper way to output the error message is to use `message(conditionMessage(cond))`. — HenrikB, Mar 02 '22 at 03:26

HenrikB · Answer 1 · 2022-03-02T03:25:03.627

Author of futureverse here: This is an interesting problem.

Here's a minimal reproducible example:

boom <- function(x) {
  tryCatch(stop("boom"), error = function(cond) {
    message(1); message(cond); message(2)
  })
}  

y <- lapply(1L, FUN = boom)
## 1
## boom2

y <- future.apply::future_lapply(1L, FUN = boom)
## 1
## Error in doTryCatch(return(expr), name, parentenv, handler) : boom

We can even reproduce this with individual futures:

> y <- boom(1)
## 1
## boom2

> f <- future::future(boom(1))
> y <- future::value(f)
## 1
## Error in doTryCatch(return(expr), name, parentenv, handler) ## : boom

First of all, it turns out that it message(cond) that trigger this odd behavior. If you instead, for instance, use message(conditionMessage(cond)), it works fine.

UPDATE 2022-03-01: After asking about this on R-devel (thread 'message() and warning() circumvent calling handlers and signal the original class, e.g. an error' on 2022-03-01 (https://stat.ethz.ch/pipermail/r-devel/2022-March/081515.html)), I conclude that using message(e) where e is an error condition is incorrect and that one should use message(conditionMessage(e)).

Technical details below:

What happens is that message(cond) end up re-signalling the caught error (= cond). And, despite message() is muffling the error signal internally, it turns out that the future still detects it and takes it as a definite error.

I have a hunch what might be happening, but I cannot promise a quick solution. I'm now tracking this in https://github.com/HenrikBengtsson/future/issues/507. Until resolved, the workaround is: "avoid resignaling the error you just caught", i.e. don't call message(cond) or warning(cond) on an error condition.

Thanks a bunch for reporting this important issue.

PS. Please consider https://github.com/HenrikBengtsson/future/discussions for future discussions, because I'm only skimming StackOverflow occasionally.

Handling exceptions using tryCatch inside future.apply::future_lapply

1 Answers1