1

I am new to multicore in R and is trying out the snowfall package to test if it is possible to speed up the apply function.

Not sure what went wrong but my multi-core implementation of sfApply() is always about 2 times slower than the apply()

Any help is greatly appreciated! Thanks in advance.

Single core example - complete to about 1.3 seconds in my PC:

x=2000
y=10000
startTime <- proc.time()

randomX <- sample(1:100,x*y, replace=T)/100+0.5
randomMatrix <- matrix(NA,x,y)
randomMatrix[,] <- randomX
randomRetX <- apply(randomMatrix,2,cumprod)

endTime <- proc.time()
endTime-startTime

Snowfall Implementation - close to 3 secs in my PC:

library(snowfall)
sfInit( parallel=TRUE, cpus=4)
x=2000
y=10000

startTime <- proc.time()

randomX <- sample(1:100,x*y, replace=T)/100+0.5
randomMatrix <- matrix(NA,x,y)
randomMatrix[,] <- randomX
randomRetX <- sfApply(randomMatrix,2,cumprod)

endTime <- proc.time()

endTime-startTime
sfStop()
Name McChange
  • 2,750
  • 5
  • 27
  • 46
user1796513
  • 11
  • 1
  • 8
    There is always some overhead in running in parallel: breaking the problem into pieces, communications between master and slaves, assembling the outputs. Maybe your problem is not large enough to benefit from parallel processing. I would not consider it for a task that only takes 1.3 seconds. – flodel Nov 03 '12 at 13:36
  • @SuperDisk, with your edit you have removed important information about computation times... – flodel Nov 03 '12 at 14:06
  • @flodel That was an accident. It's back now. – Name McChange Nov 03 '12 at 14:12
  • Try increasing the array size (x=8000) and the computation speed of afApply() is still multiple of apply(). so my feeling is that it is less of a overhead issue. – user1796513 Nov 04 '12 at 00:16

0 Answers0