Stack Overflow
Stack Overflow

Questions tagged [doparallel]

R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.

453 questions
2
votes
0 answers

Issue on using R6 classes and foreach() %dopar% together

I'm having an issue on R6 classes when used with foreach() together, possibly to do with environments (I'm using Windows). Suppose that there are two R6 classes, "class1" and "class2". method1 in class1 is dependent on class2 (see example code below…
Min
  • 179
  • 9
2
votes
3 answers

How to prevent nested foreach loop using 100% CPU of all cores in R?

I’m running a nested 3 layers foreach loop but unable to prevent the codes from 100% occupying the remote server (Linux, centOS, physical cores = 14, logical cores = 56). The framework I use…
CcMango
  • 377
  • 1
  • 4
  • 15
2
votes
0 answers

When parallel processing in caret, verbose messages do not appear in the console

I've worked in R and a question arose. I've made up an R code using the train() function in caret library, and I also set the parallel process by registDoParallel() in doParallel. If I execute without parallel processing, then verbose messages arise…
Chan-ho Lee
  • 51
  • 6
2
votes
3 answers

caret train binary glm fails on parallel cluster via doParallel

I have seen there are a lot of questions around this topic already but none seems to give a satisfying answer to my problem. I intend to use caret::train() in combination with library doParallel on a Windows machine. The documentation (The caret…
Triamus
  • 2,415
  • 5
  • 27
  • 37
2
votes
1 answer

Even with tiny data: Error in mcfork() : unable to fork, possible reason: Cannot allocate memory

I've been working on a function to handle a large Corpus. In it I use the doparallel package. Everything was working fine on 50 - 100k documents. I tested on 1M documents and received the above error. However, when I go back down to a size of corpus…
Doug Fir
  • 19,971
  • 47
  • 169
  • 299
2
votes
0 answers

Issue with %dopar% "terminate called after throwing an instance of 'std::bad_alloc'"

I have written a function where I execute some code in parallel: cl <- makeCluster(nc, outfile = "") registerDoParallel(cl, nc) pred <- foreach(s = iter(seq(1L,length(dfr_missings))), .packages = c('RANN',…
deann
  • 756
  • 9
  • 24
2
votes
1 answer

r doParallel qbeta vs RcppParallel - how to use cores at full capacity

I run the following sample code to simulate values and below is snapshot of usage of 4 cores. It takes a while to use all cores at full capacity, I'd like to understand what's going on and ultimately how to make it…
charliealpha
  • 307
  • 2
  • 12
2
votes
0 answers

How to force stop doParallel %dopar% calculations in shiny app

My shiny app does clustering using %dopar% from doParallel library. There is one thing that bothers me: if an operation takes, for example, 10 minutes to calculate, but user closes the window of an app before it's done - calculations will continue…
vladli
  • 1,454
  • 2
  • 16
  • 40
2
votes
1 answer

Speeding up stringdist in R using Parallel

I have a vector of 300 sentences, and I am trying to find elementwise JW distance using the stringdist package. The execution time for the naive implementation is too high, leading me to look for ways to reduce the runtime. I am trying to leverage…
WitchKingofAngmar
  • 212
  • 3
  • 17
2
votes
2 answers

load-balancing in R foreach loops

Is there a way to modify how R foreach loop does load balancing with doParallel backend ? When parallelizing tasks that have very different execution time, it can happen all nodes but one have finished their tasks while the last one still have…
xraynaud
  • 2,028
  • 19
  • 29
2
votes
1 answer

R parallel: error "could not find function "%do%""

This is a snippet of R code (function "Kclust") that tries to perform a task in parallel using library(doParallel). result = foreach (r = rseq, .combine=c) %dopar% { K=apply(D, 1, function(v){sum(v <= r)-1}) …
Michael
  • 1,464
  • 1
  • 20
  • 40
2
votes
1 answer

Optimizing nested foreach dopar in R

I'd like input on how my code below is structured. Would like to know if it needs to be organized in a different way to execute faster. Specifically, whether I need to be using foreach and dopar differently in the nested loops. Currently, the…
Josh
  • 69
  • 1
  • 9
2
votes
0 answers

R not able to find a sourced function

I am using doParallel and foreach packages for running my code on multiple cores. My code is something like fun1 <- function(param1, param2, param3) { do something fun2(param4, param5, param6) foreach(param4 = 1:length(param1) %dopar%…
Sim101011
  • 305
  • 1
  • 13
2
votes
0 answers

foreach Error: aborting ... Error in unserialize(nodecon) : error reading from connection

Sometimes,my R instance is silently dying.I am using foreach for parallel processing.I am using 16core and 64GB ram machine.I am using following code to execute foreach. cl <- makeCluster(6,outfile=mkclog); registerDoParallel(cl, cores =…
Aashu
  • 1,247
  • 1
  • 26
  • 41
2
votes
1 answer

Reading multiple html pages with rvest in parallel

I am attempting to scrape values from a webpage using rvest in parallel with foreach and doParallel. Specifically, I am using a real estate property identifier called a TMK to retrieve the property's census tract number from the website. In the…
ndem763
  • 320
  • 1
  • 11
1 2 3
…
30 31