6

I use the following to create a local cluster from a Jupyter notebook :

from dask.distributed import Client, LocalCluster

cluster = LocalCluster(n_workers=24)
c = Client(cluster)

Is it possible to connect from another notebook when the kernel is occupied (compute operation) ?

My goal is to access to 'total_occupancy' for example.

DavidK
  • 2,495
  • 3
  • 23
  • 38

2 Answers2

4

As suggested by @moshevi you can connect to the scheduler by providing the address.

client = Client("address-of-scheduler")

Then you can use the client.run_on_scheduler command to execute operations on the remote scheduler

client.run_on_scheduler(lambda dask_scheduler: dask_scheduler.total_occupancy)

https://docs.dask.org/en/latest/futures.html#distributed.Client.run_on_scheduler

MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • Additional info: `"address-of-scheduler"` can be obtained e.g. through `cluster.scheduler._address`. – Dahn Oct 07 '21 at 14:05
1

You could connect to the running cluster:

c_diffrent_notebook = Client('127.0.0.1:8786')  # '127.0.0.1:8786' is the default

I would advice to explicitly specify the host in the original cluster and no rely on the default.

you can access the scheduler via the clients cluster:

c_diffrent_notebook.cluster.scheduler.total_occupancy
moshevi
  • 4,999
  • 5
  • 33
  • 50
  • Yes, but I'm unable to access to "cluster.scheduler.total_occupancy" – DavidK Feb 10 '20 at 13:16
  • Thanks. In my case, c_diffrent_notebook.cluster is None. (My scheduler and workers are launched inside a docker service) – DavidK Feb 11 '20 at 10:57