Split data set and pass the subsets in parallel to function then recombine the results

Question

Here is what I am trying to do using the foreach package. I have data set with 600 rows and 58000 column with lots of missing values.

We need to impute the missing values using package called "missForest" in which it is not parallel, it takes to much time to run this data at once. so, I am thinking to divide the data into 7 data sets (I have 7 cores) with the same number of rows (my lines) and different number of col ( markers). Then using %dopar% to pass the data sets in parallel to missForest?

I do not see how to divide the data into smaller data sets and pass those data sets to missForest then recombine the outputs!

I will appreciate it so much if you can show me how?

Here is a small example, form BLR package, demonstrating my problem:

library(BLR)
library(missForest)
data(wheat)
X2<- prodNA(X, 0.1)
dim(X2)                 ## i need to divide X2 to several 7 data frames (ii)

X3<- missForest(X2)

X3$Ximp  ## combine ii data frames

What you want to do seems weird. The columns are suppose to be the variables to predict the missing values, so you can have very different results by using different columns for each model. Normally, I'd split rows not columns! — Ricardo Oliveros-Ramos, Jun 27 '13 at 19:32
Dear Ricardo: thank you for your nice answer, the variables are in the rows. that is why i want to divide by the col., again thank you — hema, Jun 27 '13 at 19:55

Steve Weston · Accepted Answer · 2013-06-27T21:47:12.847

When processing a large matrix in parallel, it can be very important to only pass as much data as is needed for each of the cluster workers. This isn't an issue when using mclapply, either directly or indirectly when using doParallel on Linux. But on Windows, input data is sent to the cluster workers via socket connections, so it can be very important.

For cases like this, I use the isplitCol function from the itertools package. It creates an iterator over blocks of columns of a matrix. Using the chunks argument, you can split the matrix so that each cluster worker gets exactly one submatrix.

Here's a translation of your example into foreach which uses isplitCol to split the input matrix into 7 submatrices, thus decreasing the data sent to each worker by a factor of seven compared to auto-exporting X2 to each worker:

library(doParallel)
library(itertools)
library(BLR)
library(missForest)
ncores <- 7
cl <- makePSOCKcluster(ncores)
registerDoParallel(cl)
data(wheat)
X2 <- prodNA(X, 0.1)
X3 <- foreach(m=isplitCols(X2, chunks=ncores), .combine='cbind',
              .packages='missForest') %dopar% {
  missForest(m)$ximp
}
print(X3)
stopCluster(cl)

how to choose a number of cores for a dataframe of 36 variables and 287 000 records? thanks! — codeforfun, Jul 19 '21 at 21:31

score 2 · Answer 2 · edited Jun 28 '19 at 04:36

2

library(multicore)
n.cores <- 7
cuts <- cut(1:ncol(X2), n.cores)
X3 <- mclapply(levels(cuts), function(x) 
  missForest(X2[, cuts == x])$ximp , mc.cores=n.cores)
X3 <- do.call(cbind, X3)

cut to split the columns into 7 intervals then mclapply to send to your 7 cores. cbind them all together at the end

edit: adding my foreach implementation. note: I have never used this package before but it seems to be doing what I would expect

library(doParallel)
library(foreach)
n.cores <- 7
cuts <- cut(1:ncol(X2), n.cores)

cl <- makeCluster(n.cores)
registerDoParallel(cl)

X3 <- foreach(x=levels(cuts), .combine=cbind, .multicombine=TRUE) %dopar% { 
  library(missForest); missForest(X2[, cuts == x])$ximp 
}

edited Jun 28 '19 at 04:36

jay.sf

60,139
8
53
110

answered Jun 27 '13 at 19:23

Jake Burkhead

6,435
2
21
32

sorry i didnt use foreach. i find this way simpler and faster. currently testing/running my foreach implementation... – Jake Burkhead Jun 27 '13 at 19:36
Hi Jake: thank you for answering my question. I am trying to install multicore on my PC " Win7 and R 15.2", R keeps telling me it is not available for this version?? any idea what should i do? thanks again – hema Jun 27 '13 at 20:02
@hema The `multicore` package has never been available on Windows. And although `mclapply` is available in the `parallel` package on Windows, it doesn't run in parallel, so it won't be of much benefit to you. – Steve Weston Jun 27 '13 at 20:08
@hema I added a foreach implementation however, as I mention in the note, its the first time I've ever used foreach. It does seem to be doing what I expect (from checking colnames of X3 and X2, and randomly looking at regions of both matrices). – Jake Burkhead Jun 27 '13 at 20:13

Ricardo Oliveros-Ramos · Answer 3 · 2013-06-27T19:29:22.530

0

You have to split your matrix in ncores parts and then combine them again. Since you're using Random Forest, you can split the data at random (and do it several times to check and validate the results).

ncores = 7

split = sample(seq(ncores), size=ncol(X2), replace=TRUE) # random partitioning

X3 = foreach(i=seq_len(ncores), .combine=cbind, .inorder=FALSE) %dopar% {
  ind = which(split==i) # the selected rows for this core
  X = rbind(ind, missForest(X2[,ind])$ximp) # add the index as first row!
}

ind = X3[1,] # get all the index back
ind = sort(ind, index.return=TRUE)$ix # sort the index to recover the original row order
X3 = X3[-1,ind] # remove the index

edited Jun 27 '13 at 19:29

answered Jun 27 '13 at 19:20

Ricardo Oliveros-Ramos

4,322
2
25
42

:the OP is using `missForest` package which focuses on missing values not the randomForest – Metrics Jun 28 '13 at 20:26
@Metrics: the missForest package USES Random Forest (?missForest). Random Forest is a machine learning technique, not an R package: http://en.wikipedia.org/wiki/Random_forest – Ricardo Oliveros-Ramos Jun 28 '13 at 21:06
:I have used randomForest :) – Metrics Jun 28 '13 at 22:34

Split data set and pass the subsets in parallel to function then recombine the results

3 Answers3

Linked