2

Im running the function parLapply inside a loop and im verifying a strange behaviour. The time per iteration was increasing significantly and it didn't make much sense such an increase.

So i started clocking the functions within the cycle to see which one was taking the most time and i found out that parLapply was taking >95% of the time. So i went inside the parLapply function and clocked it as well to see if the times between inside and outside of the function match. And they did not by quite a large margin. This margin increases over time and the difference can reach seconds which makes quite an impact on the time it takes for the algorithm to complete. 

while (condition) {
      start.time_1 <- Sys.time()

      predictions <- parLapply(cl, array, function(i){
        start.time_par <- Sys.time()

        #code

        end.time <- Sys.time()
        time.taken_par<- end.time - start.time_par
        print(time.taken_par)

        return(value)
      })

      end.time <- Sys.time()
      time.taken <- end.time - start.time_1
      print(time.taken)
}

I would be expecting that time.taken would be similar to the sum of all time.taken_par. But it is not. The sum of all time.taken_par is usually 0.026 seconds while time.taken starts by being 4 times that value, which is fine, but then increases to a lot more (>5 seconds).

Can anyone explain what is going on and/or if what im thinking should happen is wrong? Is it a memory issue?

Thanks for the help!

Edit:

The output of parLapply is the following. However in my tests there are 10 lists instead of just 3 as in this example. The size of the each individual list that is returned by parLapply is always the same and in this case is 25.

[1] 11
[[1]]
          1           2           3           4           5           6           7           8           9          10          11          12          13          14 
-0.01878590 -0.03462315 -0.03412670 -0.06016549 -0.02527741 -0.06271799 -0.05429947 -0.02521108 -0.04291305 -0.03145491 -0.08571382 -0.07025075 -0.07704650  0.25301839 
         15          16          17          18          19          20          21          22          23          24          25 
-0.02332236 -0.02521089 -0.01170326  0.41469539 -0.15855689 -0.02548952 -0.02545446 -0.10971302 -0.02521836 -0.09762386  0.02044592 

[[2]]
          1           2           3           4           5           6           7           8           9          10          11          12          13          14 
-0.01878590 -0.03462315 -0.03412670 -0.06016549 -0.02527741 -0.06271799 -0.05429947 -0.02521108 -0.04291305 -0.03145491 -0.08571382 -0.07025075 -0.07704650  0.25301839 
         15          16          17          18          19          20          21          22          23          24          25 
-0.02332236 -0.02521089 -0.01170326  0.41469539 -0.15855689 -0.02548952 -0.02545446 -0.10971302 -0.02521836 -0.09762386  0.02044592 

[[3]]
          1           2           3           4           5           6           7           8           9          10          11          12          13          14 
-0.01878590 -0.03462315 -0.03412670 -0.06016549 -0.02527741 -0.06271799 -0.05429947 -0.02521108 -0.04291305 -0.03145491 -0.08571382 -0.07025075 -0.07704650  0.25301839 
         15          16          17          18          19          20          21          22          23          24          25 
-0.02332236 -0.02521089 -0.01170326  0.41469539 -0.15855689 -0.02548952 -0.02545446 -0.10971302 -0.02521836 -0.09762386  0.02044592

Edit2:

Ok i have found out what the problem was. I have an array that i initialize using vector("list",10000). And in each iteration of the cycle i add a list of lists to this array. This list of lists has size 6656 bytes. So over the 10000 iteration it doesn't even add up to 0.1Gb. However as this array start filling up the performance of the parallelization starts to degrade. I have no idea as to why this is happening as im running the script on a machine with 64Gb of RAM. Is this a known problem?

user201305016
  • 41
  • 4
  • 1
    I can't see from your current code why this would happen. Can you provide some more context on what happens to `predictions`? In your actual code is this a list that you're appending to? Often you see these incremental durations when you're e.g. `rbind`ing to a list or otherwise having to reallocate memory in ever-greater numbers. – Peter Smittenaar May 14 '19 at 08:20
  • So predictions is just a list that contains different lists with predictions from different datasets (which is the what `parLapply` returns). The only thing i do with predictions is to run through a cycle to calculate RMSE for each one but that's after i clock it. In the code i build a dataset from the original one, i do the modelling part and the predictions. However this doesn't take much time. – user201305016 May 14 '19 at 13:32
  • 1
    ok - and are the datasets in `array` of similar size? How big are they roughly (could you update your answer with output from `lapply(array, dim)`)? Are some of the datasets further on in array rather large and it's taking the parallel infrastructure a long time to dish it out to individual processes? – Peter Smittenaar May 14 '19 at 13:46
  • 1
    The datasets inside the `parLapply` are built in the moment (inside `parLapply`) from the original dataset and they all range between 700 and 850 number of rows and only 2 columns. There is not a difference in size that would justify a 5 seconds difference i think. – user201305016 May 14 '19 at 14:25
  • 1
    Thanks for clarifying. I'm out of my depth in that case, sorry! – Peter Smittenaar May 14 '19 at 14:30
  • Also the `array` argument in parLapply is not the datasets but a mask for the original dataset so i can create the different datasets. – user201305016 May 14 '19 at 14:30
  • Thanks anyway for your time! – user201305016 May 14 '19 at 14:33

0 Answers0