Stack Overflow
Stack Overflow

Questions tagged [mclapply]

mclapply is a parallelized version of lapply, it returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X.

mclapply is a parallelized version of lapply. It returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X.

136 questions
0
votes
0 answers

R: mclapply run it through a loop

I am trying to run mclappy which loops over 2000 times. I was wondering if there is a way to make it more efficient as the code breaks on AWS SageMaker after it reaches 150 The code is: mclapply(1:2500), myfunc, mc.cores=4) myfunc has the function…
Bruce Wayne
  • 471
  • 5
  • 18
0
votes
1 answer

Looping multiple listed data frames into a single function

I am trying to execute the function varipart() from the package ade4. I am trying to use the same number dataframe from each list in the different parts of the same function. I need to pass this for each set of dataframes. ########### DATA BELOW …
ecology
  • 606
  • 3
  • 9
  • 29
0
votes
0 answers

Discrete wavelet transform in R on Big data

I'm doing a discrete wavelet transform using the following code in R. My data in data.table format is fetched from a hive table in chunks and converted into a matrix and then the wavelet transform is applied as below.…
ML_Passion
  • 1,031
  • 3
  • 15
  • 33
0
votes
1 answer

mclapply conflict with vecLib

I recently changed my R BLAS framework to vecLib, which ships with Mac. After doing so, I had problems with the parallel package in R. Here is an example: library(parallel) xx1<-matrix(runif(2*70),ncol=2) mcl.test<-mclapply(1:2,function(i)…
Devin F
  • 1
  • 2
0
votes
1 answer

R converting code to run in shared memory

Following code helps to understand number of optimal clusters. set.seed(123) # function to compute total within-cluster sum of square wss <- function(k) { kmeans(df, k, nstart = 10 )$tot.withinss } # Compute and plot wss for k = 1 to k =…
add-semi-colons
  • 18,094
  • 55
  • 145
  • 232
0
votes
2 answers

How to implement lapply function in R using package "sparklyr"

I am pretty new to Spark, I have tried to look for something on the web but I haven't found anything satisfactory. I have always run parallel computations using the command mclapply and I like its structure (i.e., first parameter used as scrolling…
ciccioz
  • 75
  • 6
0
votes
1 answer

mclapply and spark_read_parquet

I am relatively new as active user to the forum, but have to thank you all first your contributions because I have been looking for answers since years... Today, I have a question that nobody has solved or I am not able to find... I am trying to…
José Ángel Fernández Segovia
  • 3
  • 1
0
votes
1 answer

List all the indices that has a error message in a nested list R

I am using mclapply to run my code in parallel. Below is a snippet of my code. When I set mc.preschedule = FALSE, it stores the error and goes to next item in the list. I want to see what all the elements in the list has a associated error…
Jason Mathews
  • 265
  • 1
  • 3
  • 13
0
votes
0 answers

Why does mclapply function in R is more efficient than Rcpp + OpenMP?

I have a function with a loop (EstimateUniques) that is parallelized with OpenMP. I suggested that multithreading should be more efficient than multiprocessing, but when I compare this function with the simple run of "mclapply", it showed lower…
Viktor Petukhov
  • 41
  • 4
0
votes
0 answers

R: mclapply/pblapply vs. lapply - use case

I am asking for help with this example. It might not be related to the pbapply package but rather to mclapply(). However, maybe someone has an idea whats going on here. Problem The only difference is the use of pblapply()/mclapply() instead of…
pat-s
  • 5,992
  • 1
  • 32
  • 60
0
votes
1 answer

Passing mclapply() a parameter from for (i in range)

I'm trying to do this: nmf.sub <- function(n){ sub.data.matrix <- data.matrix[, (index[n, ])] ## the index is a permutation of the original matrix at a 0.8 resampling proportion (doesn't really matter) temp.result <- nmf(sub.data.matrix, rank = 2,…
Yuriy Grabovsky
  • 161
  • 14
0
votes
1 answer

Efficiently/multithread run regression predict

I know there are many posts about the issues with memory consumption of mclapply but still I'm trying to see whether there's anything that can help my case. I'm fitting a random forest model to a ~600 by 60,000 (response y by variables matrix X)…
dan
  • 6,048
  • 10
  • 57
  • 125
0
votes
1 answer

Does mclapply wait if there are dependents on output?

I have the following psuedocode situation: results1 <- mclapply(stuff, myfunc1()) results2 <- mclapply(morestuff, myfunc2()) finalresults <- mclapply(results1, results2) Is the package smart enough to wait until all the threads spawned by results1…
user1357015
  • 11,168
  • 22
  • 66
  • 111
0
votes
1 answer

function tm::tm_map encounter an error

I have a VCorpus "oanc" and I want to change all the words to lower case, so I use the following function oanc1 <- tm_map(oanc, content_transformer(tolower)) But I got a warning: Warning message: In mclapply(content(x), FUN, ...) : scheduled…
llcc
  • 63
  • 4
0
votes
0 answers

Non-linearity in lapply computing in R

I have written relatively huge function(about 500 rows; including some savings of data, learning ANN and take their prediction) which output is list of data.frames but the problem is when the output is supposed to be bigger list (e.g. 30 000…
Bury
  • 527
  • 2
  • 5
  • 15
1 2 3
…
9
10