0

The example below parallelizes 64 Sys.sleep(1) instructions using the furrr package.

The package progressr is used to display the progression of the parallel execution.

This package allows to update some text details where I want to show the count of the number of terminated instructions.

But I do not know how to share a counter between the furrr process so with my 8-core cpu it shows 8/64 at max as you can see:

enter image description here

library(furrr)
library(progressr)

handlers("progress")

long_process <- function(items) {

  nb_of_cores <- parallel::detectCores()
  plan(multisession, workers = nb_of_cores)

  N <- length(items)
  p <- progressr::progressor(along = items)

  n <- 0

  item_process <- function(item) {
    n <<- n + 1
    p(paste(n, "/", N))
    Sys.sleep(1)
  }

  future_walk(items, item_process)
}

with_progress(long_process(1:64))
pietrodito
  • 1,783
  • 15
  • 24
  • Sharing objects between parallel process is not possible without major(!) engineering, e.g. synchronization locks, mutexes, ... and if you'd try to go down that path, it's a painful experience and rarely worth your time. – HenrikB Jul 09 '23 at 10:28
  • The **progressr** package keeps track of the overall progress for you. In your example, it already reports on `8 / 64` - isn't that exactly the information you're interested in? – HenrikB Jul 09 '23 at 10:30
  • The report here 8/64 is wrong because it shows the counter of only one of the sub-process. Somehow progressr does the job since the progress bar and the percentage of progression are accurate. But I want to be able to show the absolute number and the percentage. – pietrodito Jul 09 '23 at 10:35
  • "The report here 8/64 is wrong because it shows the counter of only one of the sub-process." => I think you're missing something. **progressr** will definitely report an all iterations, i.e. 1/64, ..., 64/64. You need to set up the format of your **progress** handler to show that. I'll post an answer with an example showing that. – HenrikB Jul 10 '23 at 08:50

1 Answers1

2

Example showing how to customize the progress package handler taken from https://progressr.futureverse.org/#further-configuration-of-progress-handlers:

library(furrr)
library(progressr)

## Use the 'progress' package with custom format output for
## progress reporting, 
handlers(handler_progress(
  format = ":spin :current/:total [:bar] :percent"
))
  
## Parallelize on the local machine using all available cores
## Comment: Avoid putting this inside functions; leave it to
## the end-user to control this
plan(multisession)


long_process <- function(items) {
  p <- progressr::progressor(along = items)
  item_process <- function(item) {
    p()
    Sys.sleep(1)
  }
  future_walk(items, item_process)
}

with_progress(long_process(1:64))

This gives a progress bar in the terminal that looks like:

/ 19/64 [============>------------------------------]  30%

Note that progress is superseded by the cli package. To achieve the same using the cli handler, use:

handlers(handler_cli(
  type = "tasks",
  format = "{cli::pb_spin} {cli::pb_current}/{cli::pb_total} {cli::pb_bar} {cli::pb_percent}"
))

That will render like:

⠼ 19/64 ■■■■■■■■■                       30%
HenrikB
  • 6,132
  • 31
  • 34
  • My original problem was to display the `current/total` ratio in the `detail` part of the `withProgressShiny` function which is updated when you call `p("3/64")`. I try to put it simply in a question but do you have a quick tip or do you prefer that I ask another question? – pietrodito Jul 10 '23 at 13:57
  • "My original problem was to display the current/total ratio in the detail part of the withProgressShiny": I see. So that's related to [`progressr::handler_shiny()`](https://progressr.futureverse.org/reference/handler_shiny.html), which is a light-weight adaptor to [`shiny::withProgress()`](https://shiny.posit.co/r/reference/shiny/latest/withprogress). Ideally, `shiny::withProgress()` would have built-in support for this. You could post to a feature request to for adding some type of `format` argument support to `progressr::handler_shiny()`. – HenrikB Jul 10 '23 at 18:20