How can I send to a remote dask-distributed cluster objects whose source code only exists locally?

Question

I have a remote dask-distributed cluster to which I want to send a series of objects to be used during computations. The problem is the source code that defines the classes of those objects only exists locally and, as a consequence, pickling does not work. Is there another way that those objects could be sent without having to previously move the code into the cluster?

Answer 1

You can use Client.upload_file to push out small source files to your workers.

Otherwise you will need to ensure that the software environment between your client and workers are the same. Usually people do this by relying on a network file system, or on Docker images.