We are trying Dask Distributed to make some heavy computes and visualization for a frontend.
Now we have one worker with gunicorn that connects to an existing Distributed Dask cluster, the worker uploads the data currently with read_csv
and persist into the cluster.
I've tried using pickle to save the futures from the persist dataframe, but it doesn't work.
We want to have multiple gunicorn workers, each with a different client connecting to the same cluster and using the same data, but with more workers each one uploads a new dataframe.