I have a function to do computation, here is a simple one as an example,
def add(a,b):
return a+b
And then I execute this function 100 times in a embarrassingly parallel way,
output = [delayed(add)(i,i+1) for i in range(100)]
compute(*output)
My question is the above code will start 100 process? If so, is there a way to make it start say 10 process, so to reduce the time to start process?
The default schedulers start up thread pools or process pools with the same number of threads/processes as you have logical cores on your machine (though this is configurable). Dask won't spin up as many processes as you have tasks; as you suggest, this would be very inefficient.
You can learn a bit more about Dask's schedulers at this doc page.
The default get for dask delayed is a thread pool with as many threads as you have cores.
You can use a different get (rather than the threaded one) as well as specify get parameters via compute. To use a thread pool with 10 threads you might do dask.compute(*output, num_workers=10). To use the multiprocessing module based get with 10 workers, you might do dask.compute(*output, get=dask.multiprocessing.get, num_workers=10). (Note that the multiprocessing module, on which dask.multiprocessing is based, is not the biggest gem in the stdlib. Dask's use is probably a rare one that should mostly work if you use it in a really plain way, but if I needed process-based parallelism with dask, I'd be quick to use Distributed, even on a single host.)
