Stack Overflow
Stack Overflow

Questions tagged [doparallel]

R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.

453 questions
4
votes
3 answers

Option "cores" from package doParallel useless on Windows?

On a Linux computer, following doParallel's vignette, I use doParallel::registerDoParallel() and then I use options(cores = N) where N is the number of cores I want to use with foreach. I can verify with foreach::getDoParWorkers() that when I…
F. Privé
  • 11,423
  • 2
  • 27
  • 78
4
votes
1 answer

foreach, doParallel and random generation

Consider the very basic (and inefficient) code using parallel foreach for generating random values: cl <- makeCluster(2) registerDoParallel(cl) foreach(i = 1:100) %dopar% rnorm(1) Is it correct or are there any additional steps needed for random…
Tim
  • 7,075
  • 6
  • 29
  • 58
4
votes
0 answers

foreach partialPlot randomForest

Trying to use foreach to plot the partial dependence plot from the randomForest package. Getting error: Error in { : task 1 failed - undefined columns selected library(randomForest) library(DoParallel) library(mlbench) data(Sonar) …
Scott
  • 642
  • 7
  • 16
4
votes
2 answers

doParallel and foreach fails to parallelise merge operation

I am trying to merge a large data.frame with a small one, and parallelise the computation. The code below works perfect, maximising all cores of my machine: len <- 2000000 set.seed(666) dat = paste(sample(letters, len, rep = T), sample(0:9, len, rep…
Audrey
  • 212
  • 4
  • 15
3
votes
0 answers

How to call system() within %dopar% iterations in R

How should I call external programs from sub-instances of parallelized R? The problem could occur also on other contexts, but I am using library(foreach) and library(doFuture) on slurm-based HPC. As an example, I have created a hello.txt that…
Imsa
  • 69
  • 4
3
votes
1 answer

r markdown generates duplicates if run in parallel

I am generating several reports via r markdown. If I do them one by one - everything is okay. If I use %do% - also okay. If I use %dopar% - 3 options: Sometimes it's okay. Sometimes reports have different names but same content. Sometimes pandoc…
Maxim
  • 301
  • 1
  • 9
3
votes
0 answers

`doParallel` vs `future` while using `Seurat` package

Here is the story. From Seurat vignette, FindMarkers() can be accelerated by utilizing future package, future::plan("multiprocess", workers = 4) However, I am running a simulation that I need to use FindAllMarkers() inside a doParallel::foreach()…
yuw444
  • 380
  • 2
  • 10
3
votes
1 answer

NULL value passed as symbol address error in foreach loop R

I have never had problems with this before, but I am running into this error when trying to use a foreach loop in R: "Error in { : task 1 failed - "NULL value passed as symbol address". It is next to impossible for me to produce a small,…
user8229029
  • 883
  • 9
  • 21
3
votes
1 answer

Showing progress_bar with doParallel + foreach

I am using the example code posted here to show a progress_bar (from the progress package) with doParallel + foreach. Solutions there however make use of doSNOW (e.g. code by Dewey Brooke that I am using for testing), which is more outdated than…
elcortegano
  • 2,444
  • 11
  • 40
  • 58
3
votes
1 answer

R foreach: Read and manipulate multiple files in parallel

I have 500 tar.xz files containing 2000 csv files. I need to untar them a few tar files at a time (because of disk space), process them into a data.table, delete the csv files from disk and then save the result as RDS before moving on to the next…
HCAI
  • 2,213
  • 8
  • 33
  • 65
3
votes
1 answer

Convert for loops into foreach loops

I want to make the code below more efficient by using the foreach package. I tried it for a very long time but I don't manage to get the same result as when using the for-loops. I would like to use a nested foreach-loop including parallelization...…
Andrea
  • 33
  • 3
3
votes
0 answers

parallel foreach cant find object created within the loop

this is my first question here so i hope i'm doing it right. I'm trying to run a variant of RandomForest called Geographical Regression Forest (package SpatialML). So to train the models i'm doing a foreach loop in parallel and using a sample with…
diego.idm
  • 31
  • 2
3
votes
1 answer

Foreach and doparallel instead of for loop in R

I need to speed up the for loop through multithreading. I would like to use the libraries for this: foreach and doParallel. I used these packages before but only for processes where one result table was needed. I don't know how to use them to export…
Nicolas
  • 117
  • 8
3
votes
0 answers

When using %dopar% getting Error in { : task 1 failed - "cannot open the connection"

I want to re-write some existing code I have using foreach and %dopar%. I am getting the error below when I run the complete code. The error only shows when i call the function (in line 9) - I am puzzled by the "cannot open connection". Any help is…
J.S
  • 117
  • 1
  • 8
3
votes
0 answers

In R, can you force foreach to have only side effects?

I'm parallelizing a loop that creates a relatively large dataset at each iteration. I'm using foreach::foreach() along with the doParallel backend. When I use foreach the standard way, my RAM usage blows up way before the loop is done. I would thus…
Ben
  • 429
  • 4
  • 11
1 2 3
…
30 31