1

I have a function written in python. I want to know if a code inside that function is parallelizable, can I somehow parallelize the code inside that function without making dask API calls inside that function?

I was thinking of whether dask.delayed could help me in this. But I think it parallelizes the multiple executions of a function, but I want to parallelize the inner contents of a function. Is it even possible using dask?

Dhruv Kumar
  • 399
  • 2
  • 13

1 Answers1

1

No, dask does not auto-parallelize your Python code. You will have to use Dask constructs like dask.delayed to parallelize your code.

MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • And using dask.delayed, my whole function will execute on a single node. Can it execute on multiple nodes, using dask.delayed? – Dhruv Kumar Jun 20 '18 at 15:26
  • Each dask.delayed call can be run independently on different machines, yes. I recommend reading through http://dask.pydata.org/en/latest/delayed.html or trying examples at https://mybinder.org/v2/gh/dask/dask-examples/master?filepath=delayed.ipynb – MRocklin Jun 21 '18 at 15:30