9

if I have a distributed.Client instance can I use that to shutdown the remote cluster? i.e. to kill all workers and also shutdown the scheduler?

If that can't be done using the Client instance is there another way other than manually killing each remote process?

Dave Hirschfeld
  • 768
  • 2
  • 6
  • 15

1 Answers1

9

There is no client function specifically for this.

The scheduler has a close() method which you could call using run_on_scheduler thus

c.run_on_scheduler(lambda dask_scheduler=None: 
    dask_scheduler.close() & sys.exit(0))

which will tell workers to disconnect and shutdown, and will close all connections before terminating the process. Note that this raises an error in the client, since the connection is broken without a reply. There are probably more elegant ways.

Note that the right way to do this is probably to interact with one of the deployment cluster managers. For example, LocalCluster has a user-facing close() method that you can call directly.

--EDIT--

client.shutdown() is now available.

mdurant
  • 27,272
  • 5
  • 45
  • 74
  • *the right way to do this is probably to interact with one of the deployment cluster managers* – I'm writing a deployment cluster manager! So I'm wondering if I can close the cluster down simply using the `Client` instance... or if I have to otherwise keep track of the hpc job and shut it down via that route. The former is quite a lot easier so it would be nice if there were a convenience method to do so. Your trick above may just do the job though... – Dave Hirschfeld Jun 19 '18 at 23:34
  • 1
    You may want to start [here](https://github.com/dask/dask-jobqueue/blob/master/dask_jobqueue/core.py#L22) and the associated docs, where the scheduler and the Cluster object are in the same process. It uses a LocalCluster with zero local workers, and calls `__exit__` to stop it. – mdurant Jun 20 '18 at 00:03
  • 1
    Looks like `Client.shutdown` (with the above semantics) is now implemented in https://github.com/dask/distributed/pull/3106 – Dave Hirschfeld Oct 02 '19 at 03:27