Environment/scoping in furrr: nesting get() in future_map()

Question

I'm interested in learning more about how furrr finds stuff from the global environment, and asked generally about the black magic it performs. Here's a specific example of a behavior I didn't understand and could use some help with: What do I need to change in either in the future_map call or in the get call to return "C" and "F"?

# load furrr, describe "plan"
library(furrr)
nc<-2
plan(strategy = multiprocess, workers = nc)

# create objects

a<-list("A", "B", "C")
b<-list("D", "E", "F")


#works fine
future_map(1:5, function(foo){
    map(c("a", "b"), function(my_object_name){
        bar<-my_object_name
        print(bar)

    })
})


# object 'a' not found

future_map(1:5, function(foo){
        map(c("a", "b"), function(my_object_name){
            bar<-get(my_object_name)[[3]]
            print(bar)
    })
})

EDIT

It seems like this issue is not reproducible on all systems and may have to do with my installation of furrr. Despite the warning the package gives about multicore plans, this is an issue with multiprocess and multisession but not plan(strategy=multicore,....

score 0 · Answer 1 · answered Mar 21 '20 at 18:23

0

It is the envir that is creating the problem. Specify the envir as the global environment to look for that object and print

library(furrr)
future_map(1:5, function(foo){
    map(c("a", "b"), function(my_object_name){
        bar <- get(my_object_name, envir = .GlobalEnv)[[3]]
        print(bar)
    })
 })
#[1] "C"
#[1] "F"
#[1] "C"
#[1] "F"
#[1] "C"
#[1] "F"
#[1] "C"
#[1] "F"
#[1] "C"
#[1] "F"
#[[1]]
#[[1]][[1]]
#[1] "C"

#[[1]][[2]]
#[1] "F"
#...

answered Mar 21 '20 at 18:23

akrun

874,273
37
540
662

I had tried that without success. `> future_map(1:5, function(foo){ + map(c("a", "b"), function(my_object_name){ + bar<-get(my_object_name, envir=.GlobalEnv)[[3]] + print(bar) + }) + })` `Error in get(my_object_name, envir = .GlobalEnv) : object 'a' not found` – Michael Roswell Mar 21 '20 at 18:50
@Michael I assumed that you have created `a<-list("A", "B", "C"); b<-list("D", "E", "F")` in your global env (as showed in your post) – akrun Mar 21 '20 at 18:52
@Michael Please check your `ls()` for objects 'a', 'b', in case you are doing this on a fresh R session without that objects created – akrun Mar 21 '20 at 18:53
I believe I have: `ls() [1] "a" "b" "nc"` – Michael Roswell Mar 21 '20 at 18:54
@Michael I also have that object `ls()# [1] "a" "b" , ...` – akrun Mar 21 '20 at 18:55
@Michael Can you try. this outside `get("a", envir = .GlobalEnv)` may be there is some issue in your environment – akrun Mar 21 '20 at 18:56
@Michael also you can check the `ls` from inside the `map` `future_map(1:5, function(foo){ map(c("a", "b"), function(my_object_name) ls(envir = .GlobalEnv))})` – akrun Mar 21 '20 at 18:59
Most interesting. `get("a")` returns the expected list. `ls()` inside the `map` call did not: `[1] "future.call.arguments" "has_future" "is_bad_rlang_tilde" "version"`. Know what this means? – Michael Roswell Mar 21 '20 at 19:59
@Michael For me, it is giving that 'a'. It could mean that your version of `furrr` would be one issue – akrun Mar 21 '20 at 20:00
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/210071/discussion-between-michael-and-akrun). – Michael Roswell Mar 21 '20 at 20:10

Michael Roswell · Accepted Answer · 2020-03-23T13:22:44.060

I think I tripped across some known weird behavior with the future package with documented workarounds. See vignette in future documentation.

To add variables to the exported globals, use the globals argument in future, which translates in furrr to , .options = future_options(globals(structure=T, add="missing_object"

I suspect this might have also been one of my problems:

...The above error occurs because, contrary to the master R process, the R worker that evaluated the future expression does not have data.table loaded. Instead the evaluation falls back to the [.data.frame method, which is not what we want.

Until the future framework manages to identify data.table as a required package (which is the goal), we can guide future by specifying additional packages needed...

Environment/scoping in furrr: nesting get() in future_map()

2 Answers2