0

I am experimenting for the first time with the furrr package, especially with the future_map group of functions.

I'm having trouble understanding how to avoid running into the problem of non-exportable references even with extremely basic functions. Here is a toy example:

my_sum = function(x, a){
    return(x+a)
}
options(future.globals.onReference = "error")

library(furrr)
plan(multiprocess)
test = future_map(.x= 1:10, .f = my_sum, a = 1)

While the function does run if I don't set the error option, there is clearly some issue in the background, as the error reads Detected a non-exportable reference (‘externalptr’) in one of the globals (<unknown>) used in the future expression. I really don't understand how to get around this, any help?

Additional info: running Windows. R is version 4.0.2, furrr is version 0.1.0, future is version 1.19.1

Violetta Zoffoli
  • 15
  • 6
  • are you working on Windows? – Edo Oct 08 '20 at 08:16
  • I don't think I understand your problem.. without `options(...)` the code runs fine... What is exactly that you're trying to achieve? – Edo Oct 08 '20 at 08:18
  • Yes, I am working on Windows. This error leads me to a series of errors in more complicated functions. What I'm trying to do is understand the functioning of (non) exportable references that lie beneath parallel programming. – Violetta Zoffoli Oct 08 '20 at 08:30
  • Then I believe you should share a reproducible example of the series of errors you get later, since that is your actual goal. – Edo Oct 08 '20 at 08:35
  • 1
    Could you provide some session info, e.g. the version of {furrr} and {future}. – F. Privé Oct 08 '20 at 08:36
  • Added additional info – Violetta Zoffoli Oct 08 '20 at 08:41
  • 1
    looks like the problem is related to `future_map`. If you would use: `library(parallel);test <- parLapply(makeCluster(detectCores()), X = 1:10, fun = my_sum, a = 1)`, you have no problem.. it may be necessary to open an issue on `furrr`..? – Edo Oct 08 '20 at 09:35
  • I can replicate this with `plan(multisession)`, which is what you get on Windows when using `plan(multprocess)`. – HenrikB Oct 09 '20 at 01:22
  • 1
    My suggestion is to only use `options(future.globals.onReference = "error")` for troubleshooting. You can also use `options(future.globals.onReference = "warning")`. The default is `"ignore"` because there are lots of false positives. – HenrikB Oct 09 '20 at 01:27
  • Thanks everybody for your answers. I was basing my question on the assumption that I misunderstood the functioning of multiprocess/multisession. I'll probably use another function that does the same thing and/or open an issue on `furrr`. – Violetta Zoffoli Oct 09 '20 at 07:58
  • 1
    @HenrikB, I was expecting NOT to have the same issue with `future.apply`: `future_lapply(1:10, my_sum, a = 1)`. But I do. Then it is not a problem related to `furrr`, is it? What's the problem here? – Edo Oct 09 '20 at 10:27
  • 1
    Henrik opened the issue for you: https://github.com/DavisVaughan/furrr/issues/168. However, since `future.apply` has the same issue, it may be related to `future`. – Edo Oct 09 '20 at 10:48
  • Thanks for checking with **future.apply**. As I write in , this is an interesting problem but it's not-critical. Scanning for non-exportable globals is a complicated problem and `options(future.globals.onReference = "error")` is at best only a clue. – HenrikB Oct 09 '20 at 17:54

0 Answers0