3

Can we create a dask-cluster with some CPU and some GPU machines together. If yes, how to control a certain task must run only on CPU machine, or some other type of task should run only on GPU machine, and if not specified, it should pick whichever machine is free.?

does dask support such type of cluster.? what is the command that controls the task to run on a specific CPU/GPU machine.?

TheCodeCache
  • 820
  • 1
  • 7
  • 27

1 Answers1

2

You can specify that a Dask worker has certain abstract resources 

dask-worker scheduler:8786 --resources "GPU=2"
dask-worker scheduler:8786 --resources "GPU=2"
dask-worker scheduler:8786 --resources "MEMORY=100e9"

and that a task consumes those resources during execution. 

processed = [client.submit(process, d, resources={'GPU': 1}) for d in data]
final = client.submit(aggregate, processed, resources={'MEMORY': 70e9})

You can use this to model machines with GPUs. Note that these terms GPU and MEMORY are just abstract terms. They could just as easily been FOO and BAR.

See documentation on worker resources for more information.

MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • Hi @MRocklin, is it possible to use dask-ssh and specify number of gpus per worker? – Foivos Jun 13 '18 at 09:24
  • @mrocklin how do we set or get the `GPU ID` for example: `GPU=2` in the above command. @manoranjan Given that I have a Nvidia GTX 1070 Ti – Asif Ali Oct 24 '18 at 08:48