I am running multiple concurrent dask jobs using dask-client submit api. It have come across this issue multiple times.
Thread dump of the specific worker shows below information.
Can some one guide me about this problem.
ts_data = dask_df[key].values.compute()
File "/agent/miniconda3/envs/py361/lib/python3.6/site-packages/dask/base.py", line 135, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/agent/miniconda3/envs/py361/lib/python3.6/site-packages/dask/base.py", line 333, in compute
results = get(dsk, keys, **kwargs)
File "/agent/miniconda3/envs/py361/lib/python3.6/site-packages/distributed/client.py", line 1999, in get
results = self.gather(packed, asynchronous=asynchronous)
File "/agent/miniconda3/envs/py361/lib/python3.6/site-packages/distributed/client.py", line 1437, in gather
asynchronous=asynchronous)
File "/agent/miniconda3/envs/py361/lib/python3.6/site-packages/distributed/client.py", line 592, in sync
return sync(self.loop, func, *args, **kwargs)
File "/agent/miniconda3/envs/py361/lib/python3.6/site-packages/distributed/utils.py", line 252, in sync
e.wait(1000000)
File "/agent/miniconda3/envs/py361/lib/python3.6/threading.py", line 551, in wait
signaled = self._cond.wait(timeout)
File "/agent/miniconda3/envs/py361/lib/python3.6/threading.py", line 299, in wait
gotit = waiter.acquire(True, timeout)
