Stack Overflow
Stack Overflow

Questions tagged [doparallel]

R package that is a “parallel backend” for the foreach package. It provides a mechanism needed to execute foreach loops in parallel.

453 questions
0
votes
1 answer

nested foreach using doparallel on chunks of data in R

I'm tring to parse JSON from huge dataset with 20: As I append data over and over the process takes lot of time. And seems like times increase exponentially by rows. Hence, I thought of dividing the data into chinks, and dealing it chunk by…
letmetype
  • 105
  • 7
0
votes
1 answer

faster way to use apply in large array (maybe involving parallelization)?

I have a large array in R, let's say, A = array(rnorm(100*100*30*30*100),dim=c(100,100,30,30,100)) I would like to find the first entry in the 4th dimension that is less than 1: first_entry = apply(A,c(1,2,3,5),function(x) min(which(x <…
Alex
  • 77
  • 1
  • 6
0
votes
1 answer

How to make parallel processing in R with empty result?

I'd like to make parallel processing in R by using packages 'doParallel' and 'foreach'. And, the idea is to make parallel only computations without any outcomes. What I've found looks like 'foreach' operator always return some kind of result that…
Andrii
  • 2,843
  • 27
  • 33
0
votes
1 answer

poor performance of workers in foreach package of R

I am working with foreach and doParallel package in Windows, but the CPU used in the code is less than 10% during the foreach function. This is the code that I use with a small example.…
LauC
  • 55
  • 5
0
votes
2 answers

Using foreach function to parallelise calculation

I have a folder that has 5000 csv files, each file belonging to one location and containing daily rainfall from 1980 till 2015. Sample structure of a file is as follows: sample.file <- data.frame(location.id = rep(1001, times = 365 * 36), …
89_Simple
  • 3,393
  • 3
  • 39
  • 94
0
votes
1 answer

How to shut down an open R cluster connection using parallel

In the question here, the OP mentioned using kill to stop each individual processes, well because I wasn't aware that connections remain open if you you push "stop" while running this in parallel in R Studio on Windows 10, and like a fool I tried to…
JoeTheShmoe
  • 433
  • 6
  • 13
0
votes
0 answers

Parellize xgboost modelling simulations

I have to run multiple simulations of the models. I am trying to run different simulations on different cores and save the results to global env. I have the below code which isn't working. For example I have to run xgboost three times ( with all…
Learner_seeker
  • 544
  • 1
  • 4
  • 21
0
votes
0 answers

Speeding up R script with parallel processing

I'm using the caret package to train on a 14x21 matrix. The below script takes 65 seconds to process, which is awfully slow. I was wondering if there is any efficiency gains by utilizing the parallel packages in R? The PC I am running the code on…
ZJAY
  • 2,517
  • 9
  • 32
  • 51
0
votes
0 answers

Gower's Distance Parallel Computation in R

I am trying to compute the Gower's distance between observations of a single dataset. I found a useful function at the following link: https://www.kaggle.com/olivermeyfarth/parallel-computation-of-gower-distance?scriptVersionId=46656 which contains…
AyeTown
  • 831
  • 1
  • 5
  • 20
0
votes
1 answer

Does it make sense do distribute n tasks over n+1 cores when parallel computing? R

I just started computing in Parallel in R to optimise my code, and although I understand the basic concept, I have a seemingly simple question that I haven't found the answer to. If I have a foreach loop of n iterations and m>n cores available on my…
Lance
  • 125
  • 1
  • 7
0
votes
1 answer

indirect indexing/subscripting inside %dopar%

I'm not understanding how to do indirect subscripting in %dopar% or in llply( .parallel = TRUE). My actual use-case is a list of formulas, then generating a list of glmer results in a first foreach %dopar%, then calling PBmodcomp on specific pairs…
tom 2
  • 334
  • 2
  • 3
0
votes
1 answer

Re: couldn't find function in R even when I've installed the packages

I have this code for my paper and it still could not find the function even when I've properly re-installed the necessary packages. expoUtility <- function(x, alpha, param_beta, W){ (1-exp(-alpha*(W +…
Paolo
  • 1
  • 3
0
votes
1 answer

R: doSNOW/foreach create list of list

Hi I would like to create a named list of list using the doSNOW/foreach package. For example the end product would be a list object. dfe named from a vector say, n=c("n1","n2","n3","n4","n5") so that I can access the list of list object like…
Ahdee
  • 4,679
  • 4
  • 34
  • 58
0
votes
1 answer

Foreach save multiple classes of data R

I want to save multiple classes in a list within a foreach loop in R. I have a reproducible example here. cl <- makeCluster(4) registerDoParallel( cl) start_time <- Sys.time() acomb <- function(...) abind(..., along=3) results <-foreach(i =…
user3978632
  • 283
  • 4
  • 17
0
votes
0 answers

Processes become zombies while r parallel session still working

I'm trying to query my DB large number of times and to activate some logic over the query's result set. I'm using Roracle and dopar in order to do so (BTW-my first try was with RJDBC, but I switched to Roracle because I got Error reading from…
staove7
  • 560
  • 5
  • 18
1 2 3
…
30 31