Stack Overflow
Stack Overflow

Questions tagged [streamz]

7 questions
5
votes
1 answer

Dask: Submit continuously, work on all submitted data

Having 500, continously growing DataFrames, I would like to submit operations on the (for each DataFrame indipendent) data to dask. My main question is: Can dask hold the continously submitted data, so I can submit a function on all the submitted…
gies0r
  • 4,723
  • 4
  • 39
  • 50
5
votes
1 answer

Streamz/Dask: gather does not wait for all results of buffer

Imports: from dask.distributed import Client import streamz import time Simulated workload: def increment(x): time.sleep(0.5) return x + 1 Let's suppose I'd like to process some workload on a local Dask client: if __name__ == "__main__": …
daniel451
  • 10,626
  • 19
  • 67
  • 125
1
vote
0 answers

Streamz + Dask worker occupancy is lower than I'd expect

I've got a Dask cluster with 32 workers running on a local machine, and have tried to run the following Streamz workflow against it: I'm only seeing a couple of the workers occupied at any given time: I see increased occupancy when running locally…
sgccarey
  • 492
  • 2
  • 6
  • 16
1
vote
1 answer

Streaming for Xarray NetCDF Files

I was wondering if there was a way to stream data directly from a NetCDF file as it's being written with `xarray. I think I can "create" a non-buffered file like this? import io ts_file_stream = io.open("/some/file/being/written/to.nc", mode='rb',…
pgierz
  • 674
  • 3
  • 7
  • 14
0
votes
0 answers

No data when trying to make a real-time Dash application with Holoviews

I'm trying to create a real-time data application to monitor sensors, by displaying the last 1000 data points with a 100ms refresh. I've been able to create a Dash web app using the extendData callback on a Scattergl Plotly plot, but it uses 50% of…
Goulouboudou
  • 3
  • 2
0
votes
0 answers

Set the size of a PeriodicDataFrame

I'm using hvPlot and streamz to display real-time data from a Redis cache, in a Panel application. hvPlot is appending the datapoint to the chart, at one point when a limit of some kind of buffer is reached and the data will slide on his own,…
Goulouboudou
  • 3
  • 2
0
votes
1 answer

Correct use of streamz with websocket

I am trying to figure out a correct way of processing streaming data using streamz. My streaming data is loaded using websocket-client, after which I do this: # open a stream and push updates into the stream stream = Stream() # establish a…
SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46