1

I have a Pyramid web-application on which the customer would like the ability to plot large data sets interactively.

The application currently displays a sub-set of the customer's selected data with zoom, pan, hover, etc. capabilities using D3. However, in the case the user needs to see the full set with the same functionality, I would like to use the Bokeh server and use down-sampling.

Where I am running into trouble is that the down-sampling function can only be used with plots employing a ServerDataSource.

Ideally, the Bokeh server would be running constantly and I could push the customer's selected data to it and then use it as the source for the down-sampled plot. However, as far as I can tell, Blaze doesn't allow me to push data to an existing server.

Instead, I thought when the user requested the plot, I could use one of Pyramid's views to modify the Blaze configuration file and then start the Bokeh server. 

@view_config(route_name='startBokehServer', renderer = 'json',permission='view')
def startBokehServer_view(request):

    r"""

    Configures blaze_config.py file to create a dict containing customer's select reversal data
    Starts Bokeh server using modified blaze_config.py file at http://localhost:5006

    """
    bokeh_server.run()

Once the data was stored on the server, another view would plot the curve with the Bokeh server as the data source.

@view_config(route_name='fetchPlotDataRev', renderer = 'json',permission='view')
def fetchPlotDataRev_view(request):

  r"""

  Plots full reversal data using Bokeh's down-sampling method.

  """
    from bokeh.plotting import output_server, figure, show
    from bokeh.models import HoverTool, Range1d
    from bokeh.transforms import line_downsample      

    output_server("Full Reversal Curve(s)")
    c=bokeh_client('http://localhost:5006')
    d=bokeh_data(c)

    rev=d.rev

    source=line_downsample.source()
    source.from_blaze(rev,local=True)

    TOOLS="pan,wheel_zoom,box_zoom,reset,hover"
    p = figure(title="Reversal with tooltip", tools=TOOLS)

    p1=p.scatter(x='time',y='aa_cd',color='#0000ff',source=source)
    p1.select(dict(type=HoverTool)).tooltips=[("(x,y)", "($x, $y)"),]

    p2=p.scatter(x='time',y='aa_cv',color='#ff0000',source=source)
    p2.select(dict(type=HoverTool)).tooltips=[("(x,y)", "($x, $y)"),]

    xmin=float(rev.time.min())
    xmax=float(rev.time.max())
    ymin=float(rev.aa_cv.min())
    ymax=float(rev.aa_cd.max())

    p.x_range=Range1d(start=xmin,end=xmax)
    p.y_range=Range1d(start=ymin,end=ymax)        

    show(p)

Finally, a signal from the JavaScript that the Bokeh server window has been closed posts a request to another view that would stop the server.

@view_config(route_name='stopBokehServer', renderer = 'json',permission='view')
def stopBokehServer_view(request):

    r"""

    Stops Bokeh server.

    """
    bokeh_server.start.stop()

However, the application exits with status 2 when trying to serve /startBokehServer. The traceback is included below.

usage: pserve [-h] [--ip IP] [--port PORT] [--url-prefix URL_PREFIX]
      [-D BLAZE_CONFIG] [-m] [--script SCRIPT] [--backend BACKEND]
      [--redis-port REDIS_PORT] [--start-redis] [--no-start-redis]
      [--ws-conn-string WS_CONN_STRING] [-d] [--dev] [--filter-logs]
      [-j] [-s] [--robust-reload] [-v]
pserve: error: unrecognized arguments: development.ini
/home/katmeg/mat-cruncher/PyramidAnalysis
2015-04-09 12:20:53,730 ERROR [waitress][Dummy-11] Exception when serving /startBokehServer
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.9-py2.7.egg/waitress/channel.py", line 337, in service
task.service()
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.9-py2.7.egg/waitress/task.py", line 173, in service
self.execute()
  File "/usr/local/lib/python2.7/dist-packages/waitress-0.8.9-py2.7.egg/waitress/task.py", line 392, in execute
app_iter = self.channel.server.application(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/router.py", line 242, in __call__
response = self.invoke_subrequest(request, use_tweens=True)
  File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/router.py", line 217, in invoke_subrequest
response = handle_request(request)
  File "/usr/local/lib/python2.7/dist-packages/pyramid_debugtoolbar-2.3-py2.7.egg/pyramid_debugtoolbar/toolbar.py", line 178, in toolbar_tween
response = _handler(request)
  File "/usr/local/lib/python2.7/dist-packages/pyramid_debugtoolbar-2.3-py2.7.egg/pyramid_debugtoolbar/panels/performance.py", line 57, in resource_timer_handler
result = handler(request)
  File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/tweens.py", line 21, in excview_tween
response = handler(request)
  File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/router.py", line 163, in handle_request
response = view_callable(context, request)
  File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 245, in _secured_view
return view(context, request)
  File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 355, in rendered_view
result = view(context, request)
  File "/usr/local/lib/python2.7/dist-packages/pyramid-1.5.1-py2.7.egg/pyramid/config/views.py", line 501, in _requestonly_view
response = view(request)
  File "/home/katmeg/mat-cruncher/PyramidAnalysis/pyramidanalysis/views.py", line 649, in startBokehServer_view
bokeh_server.run()
  File "/home/katmeg/pyrEnv/local/lib/python2.7/site-packages/bokeh/server/__init__.py", line 134, in run
args = parser.parse_args(sys.argv[1:])
  File "/usr/lib/python2.7/argparse.py", line 1691, in parse_args
self.error(msg % ' '.join(argv))
  File "/usr/lib/python2.7/argparse.py", line 2347, in error
self.exit(2, _('%s: error: %s\n') % (self.prog, message))
  File "/usr/lib/python2.7/argparse.py", line 2335, in exit
_sys.exit(status)
SystemExit: 2

Note: The plots work as I intend them to when I run the bokeh-server executable from the command line and then create and serve them from a separate Python script.

So my questions are as follows: Can I push data to an already running Bokeh server and then use it as a data source for down-sampling? And if not, how can I start/stop the Bokeh server as requested within the Pyramid application?

Thanks in advance for any help!

katmeg
  • 85
  • 1
  • 10

1 Answers1

1

Here is how the application dynamically configures the Server data source, starts the Bokeh server, and plots the requested data on the the Bokeh server for the customer to navigate.

The data to be plotted is passed to the startBokehServer view where it is written to a temporary text file. The Bokeh server is then started using Python's subprocess and a configuration file that reads the text file and configures the data such that it can be read by Blaze.

views.py

@view_config(route_name='startBokehServer',renderer = 'json',permission='view')
def startBokehServer_view(request):

    # plotDataRev is dictionary containing data requested from web page
    with open('pyramidanalysis/temp/reversal_data.txt','wb') as handle:
      pickle.dump(plotDataRev,handle)  
    bokeh_pid=str(subprocess.Popen(['/bokeh-server','-D','blaze_config.py'],stdout=subprocess.PIPE).pid)
    request.session['bokehPID']=bokeh_pid

blaze_config.py

import pandas as pd
import pickle

with open('pyramidanalysis/temp/reversal_data.txt','rb') as handle:
  b=pickle.loads(handle.read())

data=dict()

keys=b.keys()
for key in keys:
  data[key]=pd.DataFrame(b[key])

The server page is then displayed in an iFrame on the web-page and closing the frame or the main window initiates another view that terminates the Bokeh server and deletes the temporary text file.

katmeg
  • 85
  • 1
  • 10