Increasing mc.cores beyond the number of logical cores

Question

Playing around with R function parallel::mclapply, I found that argument mc.cores can be chosen greater than the number of logical cores (as indicated by parallel::detectCores), resulting in speedup greater than the number of logical cores. Here's a minimal example (for me, this worked on MacOS and Linux):

sleepy <- function(i) {
    start <- Sys.time()
    Sys.sleep(i)
    as.numeric(Sys.time() - start)
}

mc.cores <- 100L
ntasks   <- 10000L

start <- Sys.time()
out <- parallel::mclapply(2/ntasks*runif(ntasks), sleepy, mc.cores = mc.cores)

real_duration <- as.numeric(Sys.time() - start)
cpu_duration <- sum(unlist(out))

data.frame(logical.cores = parallel::detectCores(),
           mc.cores      = mc.cores,
           speedup       = cpu_duration/real_duration)


##   logical.cores mc.cores  speedup
## 1             8      100 30.49574

I also tried this out in an more realistic example, i.e. close to the real scenario I want to parallelize: this didn't result in any problem, either.

In the documentation of / tutorials on parallel::mclapply, I could not find any example where mc.cores > detectCores() is chosen, and most probably, there's a very good reason for it.

Could somebody explain what are the problems with this practice? Can it be reasonable in certain circumstances, e.g. when memory requirements are not an issue?

Answer 1

I sometimes use mc.cores > detectCores() to throttle memory usage. If you split a job into 10 parts and process them with mclapply and mc.preschedule=F, each core will only process 10% of your job at a time. If mc.cores was set to two, for example, the other 8 "nodes" would have to wait until one part finished before starting a new one. This can be desirable if you're running into memory issues and want to prevent each loop from taking on more than it can handle.

Answer 2

All it does is spawning threads, you can think of them as lightweight processes with shared memory. Usually, it is not optimal to spawn more threads than cores available because of the context switching overhead. As a rule of thumb, most of the time you'll be best off with the number of workers equal to the number of logical cores of your cpu.