Stack Overflow
Stack Overflow

Questions tagged [mclapply]

mclapply is a parallelized version of lapply, it returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X.

mclapply is a parallelized version of lapply. It returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X.

136 questions
7
votes
1 answer

Combine lists in R

What would be an easy and fast way to get from: x <- list(a1=2, b1=c(1,2), c1=1:3) y <- list(a2=5, b2=c(2,5), c2=2:4) to list(list(x$a1, y$a2), list(x$b1, y$b2), list(x$c1, y$c2)) ? Or in general: If list x and y have same length and their…
Nairolf
  • 2,418
  • 20
  • 34
6
votes
4 answers

R parallel abort all mclapply operations

Is it possible to request that parallel::mclapply() abandons all further processing asap if it encounters an error (e.g., a stop()) in any one of its processes?
ivo Welch
  • 2,427
  • 2
  • 23
  • 31
6
votes
1 answer

mclapply sendmaster error only with Rscript

I intermittently get the following error when using Rscript to call mclapply from the command-line: Error in sendMaster(try(lapply(X = S, FUN = FUN, ...), silent = TRUE)) : write error, closing pipe to the master If I run the exact same code in…
louiszya
  • 61
  • 3
6
votes
1 answer

Julia pmap performance

I am trying to port some of my R code to Julia; Basically I have rewritten the following R code in Julia: library(parallel) eps_1<-rnorm(1000000) eps_2<-rnorm(1000000) large_matrix<-ifelse(cbind(eps_1,eps_2)>0,1,0) matrix_to_compare =…
Vitalijs
  • 938
  • 7
  • 18
5
votes
0 answers

split data for embarrassingly parallel with R?

I have a big RDS file that I want to work with in parallel using R. The file takes 7.3 GB of ram when loaded. If I try to use many cores, R crashes because it runs out of memory. Is there a way to tell mclapply to use shared memory instead of making…
Ignacio
  • 7,646
  • 16
  • 60
  • 113
5
votes
1 answer

R, the environment of mclapply and removing variables

I am having trouble understanding the behave of mclapply (or maybe something else). I do something like: opt.Models = mclapply(1:100, mc.cores=20, function(i){ res = loadResult(reg, id=i) return(post.Process(res)) }) loadResult loads…
user1309940
5
votes
2 answers

how to track progress in mclapply in R in parallel package

My question is related to this question. However the question referenced above uses multicore package which was replaced by parallel. Most of the functions in the response cannot be replicated in the parallel package. Is there a way to track…
forecaster
  • 1,084
  • 1
  • 14
  • 35
5
votes
1 answer

An error in one job contaminates others with mclapply

When mclapply(X, FUN) encounters errors for some of the values of X, the errors propagate to some (but not all) of the other values of X: require(parallel) test <- function(x) if(x == 3) stop() else x mclapply(1:3, test, mc.cores = 2) #[[1]] #[1]…
orizon
  • 3,159
  • 3
  • 25
  • 30
4
votes
1 answer

parallel r with foreach and mclapply at the same time

I am implementing a parallel processing system which will eventually be deployed on a cluster, but I'm having trouble working out how the various methods of parallel processing interact. I need to use a for loop to run a big block of code, which…
Luke_radio
  • 977
  • 1
  • 9
  • 15
4
votes
1 answer

mclapply additional arguments

I have created a function DevCstat(). It takes the arguments: indat, mod, Covar,txtMat, PatCovar. indat is a list, I would like to apply the function to each element of the list. mod, Covar, txtMat, PatCovar, are objects that I would like to use…
user1375871
  • 1,199
  • 1
  • 12
  • 23
4
votes
1 answer

All jobs on one core fail with R multicore

I'm using R multicore on a long list. I invoke mclapply on the list, which makes use of 12 cores on my machine. When my list has about 1000 elements long it runs fine. When my list is longer than ~2000 elements (I'm not sure at what length this…
polarise
  • 2,303
  • 1
  • 19
  • 28
3
votes
1 answer

R: inconsistent random number generation in parallel simulation with mclapply

Problem I'm trying to implement a reproducible multicore simulation and obtain inconsistent results. Please help me explain these results and advise me of a correct way of implementing this. Note that I'm working on WSL2 (I hope there is another…
Lime91
  • 65
  • 3
3
votes
0 answers

Parallel processing in R using parallel package - not reproducible with different number of cores

I'm using the parallel package and mclapply() to parallel process simulations in R, using R Programming for Data Science (Chapter 22, Section 22.4.1) as a reference. I'm setting the seed as instructed, however, when I change the number of cores used…
bob
  • 610
  • 5
  • 23
3
votes
1 answer

R foreach: Read and manipulate multiple files in parallel

I have 500 tar.xz files containing 2000 csv files. I need to untar them a few tar files at a time (because of disk space), process them into a data.table, delete the csv files from disk and then save the result as RDS before moving on to the next…
HCAI
  • 2,213
  • 8
  • 33
  • 65
3
votes
1 answer

scheduled cores ... did not deliver results, all values of the jobs will be affected in parallel::mclapply() in R 4.0.1

I'm using parallel::mclapply() with R 4.0.1 and getting the following warning: "scheduled cores ... did not deliver results, all values of the jobs will be affected". Here the result of my investigation: inspecting the function source code, I…
lulions
  • 448
  • 4
  • 10
1
2
3
…
9 10