0

I'm trying to use dask.distributed Python API to start a scheduler. The example provided in http://distributed.dask.org/en/latest/setup.html#using-the-python-api works as expected but it does not provide insight on how to supply the options need to start Bokeh web interface.

Upon inspection of dask.distributed source code, I have understood I need to provide the Bokeh options using Scheduler(services={}). Unfortunately, I have failed on trying to find the correct dictionary format for services={}.

Below is the code for dask scheduler function.

import dask.distributed as daskd
import tornado
import threading

def create_dask_scheduler(scheduler_options_dict):
    # Define and start tornado
    tornado_loop = tornado.ioloop.IOLoop.current()
    tornado_thread = threading.Thread(target=tornado_loop.start,daemon=True)
    tornado_thread.start()
    # Define and start scheduler
    dask_scheduler = daskd.Scheduler(loop=tornado_loop,synchronize_worker_interval=scheduler_options_dict['synchronize_worker_interval'],allowed_failures=scheduler_options_dict['allowed_failures'],services=scheduler_options_dict['services'])
    dask_scheduler.start('tcp://:8786')
    return dask_scheduler

scheduler_options_dict = collections.OrderedDict()
scheduler_options_dict = {'synchronize_worker_interval':60,'allowed_failures':3,'services':{('http://hpcsrv',8787):8787}}
dask_scheduler = create_dask_scheduler(scheduler_options_dict)

The error I get is:

Exception in thread Thread-4: Traceback (most recent call last):
/uf5a/nbobolea/bin/anaconda2019.03_python3.7/envs/optimization/lib/python3.7/site-packages/ipykernel_launcher.py:18:
UserWarning: Could not launch service 'http‍://hpcsrv' on port 8787.
Got the following message: 'int' object is not callable
distributed.scheduler - INFO - Scheduler at:
tcp://xxx.xxx.xxx.xxx:8786

Help and insight is very appreciated.

Alin Bobolea
  • 3
  • 4
  • Possible duplicate of [Dask dashboard not starting when starting scheduler with api](https://stackoverflow.com/questions/54061637/dask-dashboard-not-starting-when-starting-scheduler-with-api) – Alin Bobolea May 22 '19 at 21:35

1 Answers1

1

You want

'services': {('bokeh', dashboard_address): BokehScheduler, {}}

where dashboard_address is something like "localhost:8787" and BokehScheduler is in distributed.bokeh.scheduler. You will need to read up on the Bokeh server to see what additional kwargs could be passed in that empty dictionary.

mdurant
  • 27,272
  • 5
  • 45
  • 74