0

I'm using the parallel package to get better CPU utilization and I thought it will reduce computation time significantly. But I got the opposite results, while CPU utilization got almost 100% for the 4 cores that I got, the time results indicate that using the parallel produced worst results that not using it. How can it be? Is this a problem with the package? Am I missing something else? my code is big so I can't present it here.. 

time without parallel   45 sec  1.04 min 1.5 min 6.14 min
time with parallel      1.3 min 1.7 min  2.3 min 14.5 min
number of variables      78     78       78      870
number of rows          30k     50k      70k    70k
migdal menora
  • 169
  • 4
  • 16
  • 3
    What is your code? There is significant overhead to using parallel processing, so if you're trying to parallelize many short operations, then you'll spend more time setting up the parallel processing than you save and your code will slow down. – divibisan May 02 '18 at 17:05
  • HI @divibisan, my code contains a lot of small components so if I understand you correctly for each component there is a separate parallel processing set up? If so, are you familiar with a different way to reduce the time for a lot of small components in one big script file? – migdal menora May 02 '18 at 17:17
  • 2
    I'm not really an expert, but there are a bunch of other questions with answers by more knowledgeable people which might help: https://stackoverflow.com/questions/24633605/why-parallel-code-is-slower?rq=1, https://stackoverflow.com/questions/6036120/parallel-foreach-slower-than-foreach?noredirect=1&lq=1 – divibisan May 02 '18 at 17:24
  • 2
    @migdalmenora It's hard to say without code. – Dason May 02 '18 at 17:54
  • Reading the comments, my code has several random initiations and also a lot of small code components so I understand that using parallel will not help. So what other way can I use in order to get faster calculations? – migdal menora May 02 '18 at 17:56

1 Answers1

2

Before going to parallel processing you should try to improve the single core performance. Without seeing your code we cannot give any concrete advice, but the first step should be to profile your code. Useful resources are http://adv-r.had.co.nz/Performance.html and https://csgillespie.github.io/efficientR/.

Once you have achieved good single core performance, you can try parallel processing. As hinted in the comments, it is crucial to keep the communication overhead low. Again, without seeing your code we cannot give any concrete advice, but here is some general advice:

  • Do not use a sequence of multiple parallelized steps. A single parallelized step which does all the work in sequence will have lower communication overhead.
  • Use a reasonable chunk size. If you have 10.000 tasks then don't send the individually but in suitable groups. The parallel package does that by default as long as you do not use "load balancing". If you need load balancing for some reason, then you should group the tasks into a smaller number of chunks to be handled by the load balancing algorithm.
Ralf Stubner
  • 26,263
  • 3
  • 40
  • 75