0

I am trying to run mclappy which loops over 2000 times. I was wondering if there is a way to make it more efficient as the code breaks on AWS SageMaker after it reaches 150

The code is:

mclapply(1:2500), myfunc, mc.cores=4)

myfunc has the function in it. But is there a way to split it into clusters and spin off the code in chunks. I am new to this and makeClusters is something which I am unable to understand at this point.

What I mean is prospective solution to this:

mclapply(1:150), myfunc, mc.cores=4)
mclapply(151:300), myfunc, mc.cores=4)
mclapply(301:450), myfunc, mc.cores=4)
.
.
.
.
mclapply(1850:2000), myfunc, mc.cores=4)
Bruce Wayne
  • 471
  • 5
  • 18
  • This is already the way it is done, the `mc.preschedule` (default TRUE) parameter breaks the data into chunks. If you want to use a cluster, `parLapply` is the function you want to use, and it also splits the data. – thc Aug 08 '19 at 21:09
  • Is parLapply more efficient than mcapply? How would the parLapply code look for this particular case? – Bruce Wayne Aug 08 '19 at 21:12
  • 1
    They have different uses. `mclapply` is for local multi-threading only. If you need to use a cluster, use `parLapply`. `cl <- makeCluster(... – thc Aug 08 '19 at 21:26

0 Answers0