0

I want to use the package future and its sibling future.apply in order to execute a series of operations in parallel. I was able to successfully run everything interactively but given the size of the data, I want to user the Background Jobs tab in RStudio. Running the code below in a normal (interactive) RStudio session does not raise any error and I am happy with it.

library(future)

ncores = 10

# define the number of workers
plan(future::multisession(workers = ncores))

print("Done")
#> [1] "Done"

Created on 2023-01-19 by the reprex package (v2.0.1)

Apparently, when launching the job from the Background Jobs tab in RStudio, future::multisession() fails to find the parameter ncores as specified in the environment. I thought the background job manager integrated in RStudio uses Rscript in the background and that this one looks at the same environment where the script sits but maybe I am wrong.

So, if I use the "Background Jobs" tab in RStudio and try to execute the same code, I get the following error:

Error in tweak.future(function (..., workers = availableCores(), lazy = FALSE,  : 
  object 'ncores' not found
Calls: sourceWithProgress ... eval -> plan -> do.call -> <Anonymous> -> tweak.future
Execution halted

I still think that the issue is related to what environment multisession() looks at. After a few attempts, it seems that environment() is the one I am looking for. As you can see, if I print the objects defined, ncores is there but...same error.

library(future)

ncores = 10

print(ls(environment()))

# define the number of workers
plan(future::multisession(workers = ncores, envir = environment()))

print("Done")

future_error

Any idea on how to solve this issue?

Below is the output of sessionInfo().

sessionInfo()
#> R version 4.1.2 (2021-11-01)
#> Platform: x86_64-apple-darwin17.0 (64-bit)
#> Running under: macOS Big Sur 10.16
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.0.dylib
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib
#> 
#> locale:
#> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.29   withr_2.5.0     magrittr_2.0.3  reprex_2.0.1   
#>  [5] evaluate_0.15   highr_0.9       stringi_1.7.6   rlang_1.0.6    
#>  [9] cli_3.6.0       rstudioapi_0.13 fs_1.5.2        rmarkdown_2.13 
#> [13] tools_4.1.2     stringr_1.4.0   glue_1.6.2      xfun_0.31      
#> [17] yaml_2.3.5      fastmap_1.1.0   compiler_4.1.2  htmltools_0.5.2
#> [21] knitr_1.39

Created on 2023-01-27 by the reprex package (v2.0.1)

Francesco Grossetti
  • 1,555
  • 9
  • 17

1 Answers1

0

(Author of future here)

Apparently, when using Rscript, future::multisession() fails to find the parameter ncores as specified in the environment. ...

I cannot reproduce this with R 4.2.2 on Linux. If I copy your code into a test.R file and call, I get:

$ Rscript test.R
[1] "Done"

FWIW, the recommended way to set the plan is:

plan(multisession, workers = ncores)

but I'm not sure if that makes a difference.

Maybe you're using an old version of future? What's your sessionInfo()?

HenrikB
  • 6,132
  • 31
  • 34
  • That makes me happy. Though, I am not using `Rscript test.R` from the command line. I mean I could but for the moment I am using the Background Jobs tab next to the Console in RStudio. I suppose that's kinda the same...? I confirm that with `Rscript` everything works. What about the other method? – Francesco Grossetti Jan 20 '23 at 21:05
  • I see. Maybe edit your question, because it says: "Apparently, when using Rscript, future::multisession() fails to find the parameter ncores as specified in the environment." – HenrikB Jan 20 '23 at 21:56
  • Please try the recommended way of using `plan()` and report back if that works. Please also let us know your `sessionInfo()`, which is critical for anyone trying to help to help you. – HenrikB Jan 20 '23 at 22:11
  • Did you run any further tests within the Background Jobs tab? I'd like to close the question in case. – Francesco Grossetti Jan 31 '23 at 20:13
  • Sorry, but I'd prefer if you follow up on my questions and comments (which I've asked for twice), before I spend time troubleshooting something that might not be an issue. – HenrikB Jan 31 '23 at 23:50
  • I did test it with `plan(multisession, workers = scores)` and this is the error: Error in UseMethod("tweak") : no applicable method for 'tweak' applied to an object of class "c('MultisessionFuture', 'ClusterFuture', 'MultiprocessFuture', 'Future', 'environment')" Calls: sourceWithProgress -> eval -> eval -> plan -> do.call -> Execution halted – Francesco Grossetti Feb 01 '23 at 16:14
  • I also updated my initial question so if you have comments on that, please share them. I think it is more of an RStudio problem, though you could try using the Background Jobs and also report back. This would confirm that the issue indeed belongs to RStudio. If you don't have time, I will close the question anyway as I migrated to another parallel computing infrastructure. – Francesco Grossetti Feb 01 '23 at 16:17
  • `plan(multisession, workers = ...)` should _not_ give `Error in UseMethod("tweak") ...`. That suggests you've got conflicts in your R session that may cause this. Please retry in a fresh, vanilla R session and make sure `library(future); plan(multisession, workers = 2)` works. – HenrikB Feb 01 '23 at 23:01
  • When reporting on `sessionInfo()`, please do so after attaching the relevant packages, e.g. the info on **future** and all of its dependencies are missing right now. – HenrikB Feb 01 '23 at 23:03
  • Well, the test has been conducted on a fresh session. The issue was a pointer to the arm-based R installation in macOS. I am willing to check your answer as correct anyway. – Francesco Grossetti Feb 02 '23 at 10:19
  • Are you really calling `plan(multisession, workers = 2)` and not `plan(multisession(), workers = 2)`? – HenrikB Feb 02 '23 at 18:50