R plumber with future promises: RAM usage accumulation

Question

I'm developing an app that calls R through plumber APIs and each API is embedded in a future promise (using a multisession plan). The structure of an API is something like that:

#* API1
#* @get species/<scientific_name>/API1
#* @param scientific_name:string Scientific Name
#* @serializer unboxedJSON
#* @tag test
function(scientific_name) {

Prom<-future({
  
  result <- CODE_OF_THE_PROMISE
  
  gc()
  return(list(object_to_return=result))
  
}, gc=T, seed=T)
  
gc()
return(Prom)
  
}

Despite putting gc() before returning the result and the promise, I see some RAM accumulation (a few tens or hundreds of MB per API call). Sometimes I witnessed that part of this memory was released later, but not always. I would expect that as soon as the result of the API is returned, all objects are removed and then all the RAM is released. Do you see any obvious reason why it's not the case and what could be done?

I put a reproducible example below. This would be the main code:

### Set the asynchronous coding
library(promises) ; library(future)
future::plan("multisession")

### Plumber app
library(plumber)

### Other libraries
library(terra)


### Start app 
pr <- pr("API.R")
pr %>% pr_run()

And this would be the code stored in the file API.R:

#* API1
#* @get species/<scientific_name>/API1
#* @param scientific_name:string Scientific Name
#* @serializer unboxedJSON
#* @tag test
function(scientific_name) {
  
  Prom<-future({
    
    raster_test<-rast(xmin=-180, xmax=180, ymin=-90, ymax=90, resolution=0.1, crs="+init=epsg:4326", vals=rnorm(259200, 1, 10))
    result <- sum(as.vector(raster_test))
    
    gc()
    return(list(object_to_return=result))
    
  }, gc=T, seed=T)
  
  gc()
  return(Prom)
  
}

When I run the API1 with whatever scientific_name (it does not matter), my R session takes 850MB before I execute the API and 930MB afterwards. The memory was not released in the following minutes.

If needed, this is my session info:

R version 4.2.3 (2023-03-15 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22621)

Matrix products: default

locale:
[1] LC_COLLATE=French_France.utf8  LC_CTYPE=French_France.utf8    LC_MONETARY=French_France.utf8 LC_NUMERIC=C                   LC_TIME=French_France.utf8    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] terra_1.7-3      plumber_1.2.1    future_1.32.0    promises_1.2.0.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.10       codetools_0.2-19  listenv_0.9.0     digest_0.6.31     later_1.3.0       parallelly_1.34.0 R6_2.5.1          lifecycle_1.0.3  
 [9] jsonlite_1.8.4    magrittr_2.0.3    stringi_1.7.12    rlang_1.1.0       cli_3.6.0         swagger_3.33.1    rstudioapi_0.14   ellipsis_0.3.2   
[17] webutils_1.1      tools_4.2.3       httpuv_1.6.8      parallel_4.2.3    compiler_4.2.3    globals_0.16.2   

Thanks for your help!

Answer 1

(Author of Futureverse here)

Garbage collection is always tricky - not only in R. Exactly when the garbage collector runs in R is hard to predict. Using future(..., gc = TRUE) was the correct attempt here. When using plan(multisession), the future framework will call gc() on the parallel worker immediately after the result have been collected and all variables have been wiped from the worker (all done automagically). But as you've noticed, this is not always good enough.

The multisession backend relies on persistent PSOCK cluster workers. Because they are persistent, they are around until shut down, and they will always consume memory regardless of whether they're processing tasks or not. Depending on the operating system, they might also be more or less efficient in returning de-allocated memory to the operating system.

With that said, you could try the future.callr::callr backend. It relies on transient parallel workers. Each future is evaluated in a fresh background worker that is launched on the fly and as soon as the results have been collected, the worker is shut down again. This avoids the problem of memory consumption accumulating over time.