Stack Overflow
Stack Overflow

Questions tagged [r-future]

Use this tag for questions specific to the future ecosystem in R - typically used for parallel and asynchronous processing. At the core of this ecosystem is Future API and the future package, which a larger set of packages then build upon.

94 questions
0
votes
0 answers

Future callr delay with libraries

I'm using future promises in interaction with APIs from the plumber package. To avoid RAM accumulation I use callr plan (check this post for more detail) but it creates a delay in the computing of each API (bringing some API from 0.1s to 3s, which…
Victor Cazalis
  • 11
  • 2
0
votes
0 answers

How to run R commands in parallel with controlled memory usage?

I have a large object (~50GB) that I wish to break into roughly 10 non-disjoint subsets to perform analysis on concurrently. The issue is that I have had numerous bad experiences with parallelism in R whereby large memory objects from the…
shians
  • 955
  • 1
  • 6
  • 21
0
votes
1 answer

R plumber with future promises: RAM usage accumulation

I'm developing an app that calls R through plumber APIs and each API is embedded in a future promise (using a multisession plan). The structure of an API is something like that: #* API1 #* @get species//API1 #* @param…
Victor Cazalis
  • 11
  • 2
0
votes
0 answers

Using future for async in an R package - cannot find custom functions

I have an R package and I want to use future to run stuff asynchronously. future is not able to find the functions in my package. This is a known issue in the context of golem apps. I know there is a work-around that involves assigning the functions…
Will Hipson
  • 366
  • 2
  • 9
0
votes
1 answer

Nested parallelism with R future

I'm trying to read multiple large csv files with nested parallelism with future. I have a single machine with 32 cores, and I want to set up nested parallel (5 by 6) with outer 5 process with 6 cores each. I'm trying to utilize implicit parallelism…
Matthew Son
  • 1,109
  • 8
  • 27
0
votes
1 answer

Function arguments not seen in R future framework under RStudio's Background Jobs

I want to use the package future and its sibling future.apply in order to execute a series of operations in parallel. I was able to successfully run everything interactively but given the size of the data, I want to user the Background Jobs tab in…
Francesco Grossetti
  • 1,555
  • 9
  • 17
0
votes
0 answers

R plumber with future promise: assign variable to global environment

In a R plumber script in which I use future I need to assign some variables to the global environment (so that a variable created in one API can be retrieved from another API) but it does not work as expected. I created a simple example. Here you…
Victor Cazalis
  • 11
  • 2
0
votes
0 answers

Outsource parallel computation to HPC via Slurm from local machine

I want to run a loop in parallel from my local machine, but connect it to remote machines that sit on an HPC system. Here is my approach: library("future.batchtools") library("future.apply") library("future") f <- function(x) { paste0("x = ",x ,…
Frederik Ziebell
  • 1,563
  • 1
  • 10
  • 9
0
votes
1 answer

Multisession (future) and fable

I'm trying to use the ARIMA function from the fable package. I'd like to test, using cross validation, every specification, given by the pdqPDQ data.frame rows, using a multisession plan from the future package. I will then make forecasts and later…
Rodrigo Remedio
  • 640
  • 6
  • 20
0
votes
1 answer

how can I parallelize scaling a matrix (using the Seurat package) across multiple computing nodes?

I am working on scaling and clustering a matrix of single-nucleus RNA sequencing data (genes x cells) using the R package Seurat. My data is large, containing 11,500 genes and ~1.5mil cells. Due to the size of the data, the fastest way to scale the…
mahnoorh
  • 13
  • 3
0
votes
0 answers

Issue with future and furrr, 1 processor only. R

Unfortunately, this is unlikely to be reproducible as I suspect it has to do with my setup. I'm on a windows machine, with a custom library location. Whenever I run future_map etc, only one core seems to be loaded. I suspect this is because my…
rwhit
  • 45
  • 6
0
votes
0 answers

Any way to use future.apply parallelization without PBS/TORQUE killing the job?

I frequently use the packages future.apply and future to parallelize tasks in R. This works perfectly well in my local machines. However, if I try to use them in a computer cluster, managed by PBS/TORQUE, the job gets killed for violating the…
D1X
  • 5,025
  • 5
  • 21
  • 36
0
votes
0 answers

Unable to use future_async function in R shiny app

I am trying to use the future_async() function as described here. But I get an error Warning: Error in future_async: could not find function "future_async" 51: eval [loadData.R#211] 50: eval 48: source 47: server…
dstone42
  • 1
0
votes
1 answer

no applicable method for 'offspring' applied to an object of class "phylo"

I think my issue is actually with future and exporting methods rather than tidytree, but if someone could explain to me what is actually happening and how I should be writing my code to avoid this problem it would be greatly appreciated. I am using…
august
  • 725
  • 5
  • 15
0
votes
0 answers

future plan(multisession()) error when setting workers

I am trying to use furrr to parallelize some code on a server with 144 cores available. When I run plan(multisession(), workers = 10), I get the following error: Error: Cannot create 144 parallel PSOCK nodes. Each node needs one connection but there…
ADF
  • 522
  • 6
  • 14
1 2 3 4
5
6 7