Stack Overflow
Stack Overflow

Questions tagged [doparallel]

R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.

453 questions
0
votes
1 answer

for vs lapply and time increase due to calling a function in R

I tested the following code (on a 300K row DF) to determine which is the fastest way (for loop vs lapply) for parallelization in R. Q1. Is this always true that (based on elapsed time comparisons) Parallel lapply is faster than parallel for loop?…
richard rodrigues
  • 71
  • 6
0
votes
0 answers

Rselenium in parallel: error when try to open page in a node

On following site: Run RSelenium in parallel, I have found out how to open multiply browsers (sessions) in parallel using parallel package: library(RSelenium) library(rvest) library(magrittr) library(foreach) library(doParallel) # number of…
Mislav
  • 1,533
  • 16
  • 37
0
votes
0 answers

R %dopar% can not find my model list

Hello there I am having a problem using foreach and doparallel where my loop is giving me and error that it cannot find some of my list variables. I have 3 lists in this loop that I have created seperately outside. models.list This is a list of 527…
Rhys Jevon
  • 9
  • 1
0
votes
2 answers

R memory blowup when predicting nnet output in parallel with foreach

I have a (large) neural net being trained by the nnet package in R. I want to be able to simulate predictions from this neural net, and do so in a parallelised fashion using something like foreach, which I've used before with success (all on a…
cmcsorley17
  • 146
  • 8
0
votes
1 answer

foreach and dopar not creating objects, but returning NULL

I'm using Maxent for some spatial analysis. I have a long script with many outputs that I collect in lists. It works just fine with a for loop and low resolution climatic predictors in a rasterstack (in my core i5, 6gb notebook) . But I need to use…
Thai
  • 493
  • 5
  • 15
0
votes
0 answers

using doparallel and foreach in R for nested loops and continuously printing output

I am trying to compute AUC for multiple gene expression combinations. I have ~2000 upregulated genes and ~1500 downregulated genes. I need to use parallel for my code as it is taking very long to compute. I tried to change my code a little(below)…
Sia
  • 43
  • 5
0
votes
1 answer

foreach %dopar% yielding “cannot find function” errors when the function to be parallelised is a list element

I’ve successfully parallelised a function – let’s call it AddOne - via the doParallel package, foreach and %dopar% and I’m familiar with the .packages and .export arguments to foreach. My problem is that I would like AddOne, instead of being a…
Philip Swannell
  • 895
  • 7
  • 17
0
votes
0 answers

Error in { : task 3 failed - "RcallMethod: attempt to call a method of a NULL object."

Any help of this would be greatly appreciated. I have the following R code (following tutorial at https://www.r-bloggers.com/parallel-r-loops-for-windows-and-linux/) to carry out 233 mutually exclusive calculations in parallel. There are 4 cores…
KalC
  • 1,530
  • 3
  • 22
  • 33
0
votes
0 answers

Why does doparallel::getDoParWorkers() return 14 but AFTER I have completed the loop?

My understanding was that if registering an implicit cluster with e.g. registerDoParallel(cores = 14) then at the end of the loop the cluster is automatically closed. In trying to debug another issue I wanted to see if the links to the workers (I…
Doug Fir
  • 19,971
  • 47
  • 169
  • 299
0
votes
1 answer

Optimized way to perform enrichment analysis

I am trying to perform enrichment analysis using phyper function from R. The code I have written is giving me the accurate results but it takes forever when the size of a matrix gets increased. Below is the reproducible example for 621 * 1860…
Jason Mathews
  • 265
  • 1
  • 3
  • 13
0
votes
0 answers

Cannot allocate memory even after deleting large objects and closing connections

I'm encountering an intermittent error message in R after using parallel::dopar. I use a dopar loop once after starting a fresh session and everything works. My script runs, takes a corpus, transforms it and outputs a matrix. If I rm(list = ls())…
Doug Fir
  • 19,971
  • 47
  • 169
  • 299
0
votes
1 answer

doparallel nesting a loop in a loop works but logically doesn't make sense?

I have a large corpus I'm doing transformations on with tm::tm_map(). Since I'm using hosted R Studio I have 15 cores and wanted to make use of parallel processing to speed things up. Without sharing a very large corpus, I'm simply unable to…
Doug Fir
  • 19,971
  • 47
  • 169
  • 299
0
votes
0 answers

tm_map site of transformations returning an object of NULL only on large data set

I'm working with text data and I have a function to run some standard transformations. When I test this function on a sample of 10k, 100k and even 1M the function returns the desired object, a corpus with processed text data. However, when I run on…
Doug Fir
  • 19,971
  • 47
  • 169
  • 299
0
votes
0 answers

R - caret - Print iteration when using parallel backend

I would like to generate progress printing while running a parallel back-end with the caret package. Same question and solution here, yet the solution does not work for me, any help would be appreciated: r package caret-Print iteration when using…
BigTimeStats
  • 447
  • 3
  • 12
0
votes
0 answers

foreach: code produces different results with %do% and %dopar%

As a disclaimer, this is my first post, and would I welcome any constructive criticism on appropriate postings in the future. The input to my function is an array. When I input the array using the %do% option of foreach, I get the desired output…
DeLorean
  • 1
1 2 3
…
30 31