How to create a custom Dask worker with imports

Question

I'm setting up Dask, and I can use dask for multiprocessing just fine.

I run into issues, however, when I want to use pre-configured Dask workers. They don't have the same imports I do with my main process.

I was wondering. How do I add custom imports into dask workers so all futures accessing those workers can operate effectively.

score 1 · Accepted Answer · answered Nov 03 '19 at 15:37

Ideally you Dask workers should all have the same software environment. Typically this is guaranteed outside of Dask with Docker images or with a Network File System (NFS). There are some other solutions like Client.upload_file, which can be useful for small scripts.

How to create a custom Dask worker with imports

1 Answers1