Stack Overflow
Stack Overflow

Questions tagged [doparallel]

R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.

453 questions
5
votes
1 answer

R doParallel foreach with error handling for independant workers

I have to run a lot of random forest models so I want to use doParallel on my server with 8 cores to speed up the process. Yet, some models need a lot longer than others or even might throw errors. I would like to run 8 models in parallel, and if a…
user670186
  • 2,588
  • 6
  • 37
  • 55
5
votes
2 answers

Parallel computation: Loading packages in each thread only once

I am currently working with some large datasets, so parallelizing the workflows is the only way to go. I need to load some packages to each thread once at the beginning (i.e: for(this.thread in threads) { #load some packages }. Unfortunately , I'm…
user5363218
5
votes
1 answer

Using fread with foreach and doParallel in R

I used fread with foreach and doParallel package in R 3.2.0 in ubuntu 14.04. The following code works just fine, even though I didn't use…
Lamothy
  • 337
  • 4
  • 17
4
votes
3 answers

How can I read multiple large CSVs in chunks in parallel using R?

I have 10 very large CSV files (which may or may not have the same headers) that I am reading in and processing consecutively in chunks using "readr" package read_csv_chunked(). Currently, I am able to read the 10 files in parallel using 10 cores.…
Annabanana
  • 91
  • 1
  • 3
  • 13
4
votes
1 answer

makeCluster hangs on Windows - parallel package

I have tried using the parallel package to run a simple parallelization in R on my Windows 10 laptop, but when it gets to the makeCluster function it just hangs there. I tried running on the GUI, different number of cores, changing the settings on…
Luis Henrique Linhares
  • 41
  • 2
4
votes
1 answer

Error writing to connection - Parallelization with R - Linux/Ubuntu issue

We are trying to run a code involving parallelization in R using the ResistanceGA package which calls the doParallel package. We have extremely large memory so this should not be the issue. This is the error we get: Error in serialize(data,…
Julian Wittische
  • 1,219
  • 14
  • 22
4
votes
1 answer

How can I add code between the inner and outer loops of nested foreach loops in R

I've read that the correct way to do nested foreach loops in R is via the nesting operator %:% (e.g. https://cran.r-project.org/web/packages/foreach/vignettes/nested.html). However, code can't be added between the inner and outer loops when using…
jruf003
  • 980
  • 5
  • 19
4
votes
1 answer

Foreach loop returns an error message for first set of workers only

Something very strange started happening in a package I have been working on. I am using a foreach loop with a doparallel backend to perform repeated random walks with restarts on an adjacency matrix (reproducible example…
Davis Tucker Weaver
  • 56
  • 3
4
votes
2 answers

Error in makePSOCKcluster(names = spec, ...) : Cluster setup failed. 3 of 3 workers failed to connect

I am trying to recruit more cores to increases my processing time for some lidar data I am analyzing but I keep getting "Error in makePSOCKcluster(names = spec, ...) : Cluster setup failed. 3 of 3 workers failed to connect." after I run this: …
lavp_theOG
  • 81
  • 1
  • 5
4
votes
0 answers

How to show progressbar in doParallel?

doSNOW can show progressbar like below How to show the progress of code in parallel computation in R? is there a way to get progressbar in doParallel? Update: below is a minimal reproducible example library(foreach) library(doParallel) mc <-…
Kevin Ho
  • 107
  • 1
  • 13
4
votes
1 answer

How to use `foreach` and `%dopar%` with an `R6` class in R?

I ran into an issue trying to use %dopar% and foreach() together with an R6 class. Searching around, I could only find two resources related to this, an unanswered SO question and an open GitHub issue on the R6 repository. In one comment (i.e.,…
Mihai
  • 2,807
  • 4
  • 28
  • 53
4
votes
0 answers

How to resolve %>% pipe error when running function in parallel processing?

I am running the following function in both 1 core and in parallel: the simple version seems to work fine but when loading library(parallel) it gives me an error. you should be able to run this code straight off the bat: #…
Alex Bădoi
  • 830
  • 2
  • 9
  • 24
4
votes
2 answers

Is there a way to use period.apply with doParallel and foreach in xts?

I'd like to parallelize a period.apply function in R, I'm trying to use doParallel with Foreach, but I don't know how i could implement this function. The data I'm using is an xts object with the date time index and the values of a variable, and…
Random Cotija
  • 613
  • 1
  • 10
  • 26
4
votes
1 answer

caret error if doParallel and recipes is used

When I use caret together with the new recipes package, i.e., caret::train.recipe(), then I receive an error message if also doParallel is used to register a parallel backend. Attached is a reproducible example (a short version of the official…
Oliver Pfaffel
  • 41
  • 3
4
votes
2 answers

doParallel:::doParallelSNOW complains when foreach(..., .export) is specified

I'm curious what the design argument for doParallel:::doParallelSNOW() giving a warning when globals are explicitly specified in .export could be? For example, library("doParallel") registerDoParallel(cl <- parallel::makeCluster(2L)) a <- 1 y <-…
HenrikB
  • 6,132
  • 31
  • 34
1 2
3
…
30 31