2

I've installed dask using conda. When I create delayed functions and run them over my PBS cluster using dask, how do I ensure that the worker nodes activate the same conda environment before running the delayed functions?

user2966505
  • 67
  • 4

1 Answers1

0

This document gives you good general information about how imports and environments work across a dask cluster. In your PBS case, you may fall into the case where each workers see the same filesystem.

To configure the PBS launching, you should read this, where you can define various settings, including the paths that the workers will be searching. If you also/instead have local storage on the nodes, then you might want to distribute environments using one of the methods in the first link.

mdurant
  • 27,272
  • 5
  • 45
  • 74
  • You're right, I'm using a shared filesystem, so I don't need help distributing the files. However, I do need the workers to activate the conda environment upon startup before running the python code. What startup hook can I use to get them to do that? The PBS launching link you gave doesn't seem to describe any such hook. – user2966505 Nov 10 '20 at 22:18