2

I am running a function in parallel in R using future_lapply(), but I want to print a message to the R console.

I have something like this:

library(future.apply)

plan(multisession, workers = 2)
future_lapply(1:12, function(idx) {
  Sys.sleep(5)
  print(paste0( idx, " completed."))
})

But the output of each print() is only displayed when all the processes end. How can I output them as soon as they are generated?

D1X
  • 5,025
  • 5
  • 21
  • 36
  • Try this: `future_lapply(1:12, function(idx) { Sys.sleep(idx^2); message(paste0( idx, " completed.")) }, future.conditions = "message")` – Roland Nov 05 '20 at 09:48
  • @Roland It certainly works with your example, but not with `future_lapply(1:12, function(idx) { Sys.sleep(5); message(paste0( idx, " completed.")) }, future.conditions = "message")`. Messages are still output at the end of execution of `future_lapply()` – D1X Nov 05 '20 at 09:57
  • 1
    I might misunderstand something here. But with that code, all workers finish simultaneously and afterwards, the main process has to do practically nothing. In fact, on my system even your `print` approach works fine. But using the message stream is of course better style. – Roland Nov 05 '20 at 10:01
  • @Roland **Note that I have set workers to 2 so it can be run in parallel in most computers**. As far as I understand it, since there are 2 workers, it should output the first 2 messages at approximately 5 seconds, after 10 seconds 2 more and so on. All this assuming 2 cores/threads at least are available. – D1X Nov 05 '20 at 10:04
  • Ah, okay. No, the conditions are signaled after the worker has finished. You could change the chunk sizes and, e.g., set `future.scheduling = Inf`. – Roland Nov 05 '20 at 10:12
  • @Roland That does not work either... Messages appear at the end of execution. Is not there an option that does not rely on changing chunk sizes and messing with futures? – D1X Nov 05 '20 at 10:22
  • Easiest option: `sink` to a text file. – Roland Nov 05 '20 at 10:28
  • Are you just trying to show the progression of your parallel cycle? If so, there are better ways. Have a look at the R package `progressr` – Edo Nov 05 '20 at 10:30
  • @Edo I am aware of `progressr`, but it doesn't work when launching the script and outputing to a text file. @Roland I will have a look at sink, thanks. – D1X Nov 05 '20 at 10:55
  • This seems to be closer and builds on @Roland. `future_lapply(1:6, function(idx) { Sys.sleep(idx); message(paste0( idx, " completed. ", Sys.time() )) }, future.conditions = "message", future.stdout = NA) `. It just adds `future.stdout = NA`. Note, the help suggests that using that option is not recommended. – Cole Dec 09 '20 at 01:53

0 Answers0