2

I have written a function with 3 nested foreach loops, running in parallel. The goal of the function is to split a list of 30 [10,5] matrices (i.e. [[30]][10,5]) into a list of 5 [10,30] matrices (i.e. [[5]][10,30]).

However, I am trying to run this function with 1,000,000 paths (i.e. foreach (m = 1:1000000)), and obviously, the performance is terrible.

I'd like to avoid apply functions if possible because I've found that they don't work well when used in conjunction with parallel foreach loops:

library(foreach)
library(doParallel)

# input matr: a list of 30 [10,5] matrices
matrix_splitter <- function(matr) {
  time_horizon <- 30
  paths <- 10
  asset <- 5

  security_paths <- foreach(i = 1:asset, .combine = rbind, .packages = "doParallel", .export = "daily") %dopar% {
    foreach(m = 1:paths, .combine = rbind, .packages = "doParallel", .export = "daily") %dopar% {
      foreach(p = daily, .combine = c) %dopar% {
        p[m,i]  
      }
    }
  }
  df_securities <- as.data.frame(security_paths)
  split(df_securities, sample(rep(1:paths), asset))
}

Overall, I'm trying to convert this data format:

[[30]]
            [,1]        [,2]       [,3]       [,4]        [,5]
 [1,]  0.2800977  2.06715521  0.9196326  0.3560659  1.36126507
 [2,] -0.5119867  0.24329025  0.1513218 -1.2528092 -0.04795098
 [3,] -2.0293933 -1.17989270  0.3053376 -0.9528611  0.86758140
 [4,] -0.6419024 -0.24846720 -0.6640066 -1.7104961 -0.32759406
 [5,] -0.4340359 -0.44034013  3.3440507  0.7380613  2.01237069
 [6,] -0.6679914 -0.01332117  1.9286056 -0.7194116  0.15549978
 [7,]  0.5919820  0.11616685 -0.8424634 -0.7652715  1.34176688
 [8,]  0.8079152  0.40592119 -0.4291811  0.9358829 -0.97479314
 [9,] -0.0265207 -0.03598320  1.1287344  0.4732984  1.37792596
[10,]  1.0553966  0.65776721 -1.2833613 -0.2414846  0.81528686

To this format (obviously up to V30):

$`5`
V1         V2          V3         V4         V5         V6         V7
result.2   -0.11822260  1.7712833  1.97737285 -1.6643193  0.4788075  1.2394064  1.4800787
result.7   -1.23251178  0.4267885 -0.07728632  0.3463092  0.8766395  0.6324840  0.5946710
result.2.1 -1.27309457 -0.3128173 -0.79561297 -0.4713307 -0.4344864  0.4688124 -0.5646857
result.7.1  0.51702719 -1.6242650 -2.37976199 -0.1088408  0.4846507 -0.7594376  0.9326529
result.2.2  1.77550390  0.9279155  0.26168402  0.4893835  1.4131326  0.5989508 -0.3434010
result.7.2 -0.01590682 -0.5568578  1.35789122 -0.1385092 -0.4501515 -0.2581724  0.5451699
result.2.3  0.30400225 -1.0245640 -0.05285694 -0.1354228  0.3070331 -0.7618850  1.0330961
result.7.3 -0.08139912  0.4106541  1.40418839  0.2471505  1.2106539  1.3844721  0.4006751
result.2.4  0.94977544 -0.8045054  1.48791211  1.4361686 -0.3789274 -1.9570125 -1.6576634
result.7.4  0.70449194  1.6887800  0.56447340  0.6465640  2.6865388 -0.7367524  0.6242624
                     V8         V9         V10         V11        V12         V13
result.2   -0.432404728 -1.6225350  0.09855465  0.17371907  0.3081843  0.15148452
result.7   -0.597420706  0.6173004  0.07518596  2.01741406  0.1767152 -0.39219471
result.2.1  0.918408322 -1.6896424 -0.13409626  0.38674224  0.3491750 -1.61083286
result.7.1  2.564057340 -0.7696399  1.06103614  1.38528367  1.1684045 -0.08467871
result.2.2  0.951995816  0.1910284  1.79943500  2.13909498  0.2847664  0.31094568
result.7.2 -0.479349220 -0.2368760  0.04298525 -0.40385960  0.3986555 -1.93499213
result.2.3 -1.382370069  1.0459845 -0.33106323 -0.43362925  0.7045572 -0.30211601
result.7.3 -1.457106442  0.1487447 -2.52392942 -0.02399523 -1.0349746  0.87666365
result.2.4 -0.848879365  0.7521024  0.16790915  0.47112444  0.8886361 -0.12733039
result.7.4 -0.003350467  0.4021858 -1.80031445 -1.42399232  1.0507765 -0.36193846
Jerry Zhang
  • 1,352
  • 1
  • 9
  • 20
  • 1
    how do you want to rearrange? in your example no figure of the output is present in the output. – Colonel Beauvel Sep 17 '15 at 18:18
  • it's really just from `[[30]][10,5]` to `[[5]][10,30]` – Jerry Zhang Sep 17 '15 at 18:21
  • 1
    I don't find any of this very clearly explained at all, but I suspect you might find the package (and function) **abind** helpful, followed by the function `aperm`. – joran Sep 17 '15 at 18:42
  • Is it performance suffer from the parallel overhead? You basically do nothing and call it in parallel. Beside as I understood do you want to change `[[30]][1000000,5]` into `[[5]][1000000,30]`? – Marek Sep 17 '15 at 21:29

3 Answers3

1

The package plyr is designed for this problem thanks to alply. The idea is: unlist your list, fromat it in the appropriate way in an array, and convert this array to a list of matrix using alply.

Example of transformation of a list of 2 matrix 3x5 to a list of 5 matrix 2x3:

library(plyr)

lst = list(matrix(1:15, ncol=5), matrix(10:24, ncol=5))

alply(array(unlist(lst), c(2,3,5)),3)

#$`1`
#     [,1] [,2] [,3]
#[1,]    1    3    5
#[2,]    2    4    6

#$`2`
#     [,1] [,2] [,3]
#[1,]    7    9   11
#[2,]    8   10   12

#$`3`
#     [,1] [,2] [,3]
#[1,]   13   15   11
#[2,]   14   10   12

#$`4`
#     [,1] [,2] [,3]
#[1,]   13   15   17
#[2,]   14   16   18

#$`5`
#     [,1] [,2] [,3]
#[1,]   19   21   23
#[2,]   20   22   24
Colonel Beauvel
  • 30,423
  • 11
  • 47
  • 87
  • Thank you. This is great for things at a small scale. However, when I get to 5,000,000 paths, the vector created by unlist is too large (5.6 GB). Is there any way to do this without `unlist()`? – Jerry Zhang Sep 17 '15 at 21:38
0

I believe you are looking for the answer to this: Function to split a matrix into sub-matrices in R

You would just use do.call(rbind, matlist) as an input to those functions.

Community
  • 1
  • 1
Allen Wang
  • 2,426
  • 2
  • 24
  • 48
0

I would convert all of your list into a great big vector, and then re-dimension it.

For my solution, I started with:

[[28]]
        [,1] [,2] [,3] [,4] [,5]
  [1,]    1   11   21   31   41
  [2,]    2   12   22   32   42
  [3,]    3   13   23   33   43
  [4,]    4   14   24   34   44
  [5,]    5   15   25   35   45
  [6,]    6   16   26   36   46
  [7,]    7   17   27   37   47
  [8,]    8   18   28   38   48
  [9,]    9   19   29   39   49
 [10,]   10   20   30   40   50

Repeated thirty times. This is the variable orig. My code:

flattened.vec <- unlist(orig)  #flatten the list of matrices into one big vector
dim(flattened.vec) <-c(10,150) #need to rearrange the vector so the re-shape comes out right
transposed.matrix <- t(flattened.vec) #transposing to make sure right elements go to the right place
new.matrix.list <- split(transposed.matrix,cut(seq_along(transposed.matrix)%%5, 10, labels = FALSE))  #split the big, transposed matrix into 5 10x30 matrices

This code gives you 5 vectors, that you need to dim(10,30) and then use t() on them in a foreach to get 5 30X10 vectors (I would normally use an apply function, and am not familiar with the foreach library).

End result for one of the 5 matrices result after doing so:

      [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13] [,14] [,15] [,16] [,17]
 [1,]    1    1    1    1    1    1    1    1    1     1     1     1     1     1     1     1     1
 [2,]    2    2    2    2    2    2    2    2    2     2     2     2     2     2     2     2     2
 [3,]    3    3    3    3    3    3    3    3    3     3     3     3     3     3     3     3     3
 [4,]    4    4    4    4    4    4    4    4    4     4     4     4     4     4     4     4     4
 [5,]    5    5    5    5    5    5    5    5    5     5     5     5     5     5     5     5     5
 [6,]    6    6    6    6    6    6    6    6    6     6     6     6     6     6     6     6     6
 [7,]    7    7    7    7    7    7    7    7    7     7     7     7     7     7     7     7     7
 [8,]    8    8    8    8    8    8    8    8    8     8     8     8     8     8     8     8     8
 [9,]    9    9    9    9    9    9    9    9    9     9     9     9     9     9     9     9     9
[10,]   10   10   10   10   10   10   10   10   10    10    10    10    10    10    10    10    10

        [,18] [,19] [,20] [,21] [,22] [,23] [,24] [,25] [,26] [,27] [,28] [,29] [,30]
 [1,]     1     1     1     1     1     1     1     1     1     1     1     1     1
 [2,]     2     2     2     2     2     2     2     2     2     2     2     2     2
 [3,]     3     3     3     3     3     3     3     3     3     3     3     3     3
 [4,]     4     4     4     4     4     4     4     4     4     4     4     4     4
 [5,]     5     5     5     5     5     5     5     5     5     5     5     5     5
 [6,]     6     6     6     6     6     6     6     6     6     6     6     6     6
 [7,]     7     7     7     7     7     7     7     7     7     7     7     7     7
 [8,]     8     8     8     8     8     8     8     8     8     8     8     8     8
 [9,]     9     9     9     9     9     9     9     9     9     9     9     9     9
[10,]    10    10    10    10    10    10    10    10    10    10    10    10    10

Incidentally, this is probably what the plyr package does on its own already (as posted by Colonel Beauvel), just manually instead of using an external library

Frank
  • 544
  • 2
  • 14
  • Thank you. This is great for things at a small scale. However, when I get to 5,000,000 paths, the vector created by unlist is too large (5.6 GB). Is there any way to do this without `unlist()`? – Jerry Zhang Sep 17 '15 at 21:38
  • Does the data have to come in as a list? From my testing it does seem that `unlist()` is slow (setting the `use.names=FALSE` helps a little but not by much). However, if you could start with a 3-dimensional vector instead, that inefficiency drops out (and the storage required by your data set would shrink about 10%) – Frank Sep 19 '15 at 03:31