I'm using python 3.8.5 on an Ubuntu machine. I have a function getData that makes an API call with requests.post for a specific string value which returns a pandas DataFrame converted to a list. When I iterate in serial over a list calendarCols_All of possible values for the string, the function works as expected. I'm trying to parallelize the call, using dask with the code:
import dask
from dask.distributed import Client
from dask import delayed, compute
client = Client(threads_per_worker=1, n_workers=6)
print(client)
lazy_results = [dask.delayed(getData)(col) for col in calendarCols_All]
parallelR = dask.compute(*lazy_results)
This code errors with this stack trace:
/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first
self.make_current()
<Client: 'tcp://127.0.0.1:41067' processes=1 threads=6, memory=246.61 GiB>
Traceback (most recent call last):
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/worker.py", line 2870, in dumps_function
result = cache_dumps[func]
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/collections.py", line 24, in __getitem__
value = super().__getitem__(key)
File "/usr/lib/python3.8/collections/__init__.py", line 1010, in __getitem__
raise KeyError(key)
KeyError: <function getData at 0x7f8888c773a0>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 54, in dumps
result = cloudpickle.dumps(x, **dump_kwargs)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 632, in dump
return Pickler.dump(self, obj)
File "stringsource", line 2, in grpc._cython.cygrpc.Channel.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./test_parallel_query.py", line 68, in <module>
parallelR = dask.compute(*lazy_results)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/dask/base.py", line 600, in compute
results = schedule(dsk, keys, **kwargs)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/client.py", line 3032, in get
futures = self._graph_to_futures(
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/client.py", line 2940, in _graph_to_futures
dsk = dsk.__dask_distributed_pack__(self, keyset, annotations)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/dask/highlevelgraph.py", line 1078, in __dask_distributed_pack__
"state": layer.__dask_distributed_pack__(
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/dask/highlevelgraph.py", line 432, in __dask_distributed_pack__
dsk = toolz.valmap(dumps_task, dsk)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/toolz/dicttoolz.py", line 85, in valmap
rv.update(zip(d.keys(), map(func, d.values())))
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/worker.py", line 2908, in dumps_task
return {"function": dumps_function(task[0]), "args": warn_dumps(task[1:])}
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/worker.py", line 2872, in dumps_function
result = pickle.dumps(func, protocol=4)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 58, in dumps
result = cloudpickle.dumps(x, **dump_kwargs)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py", line 632, in dump
return Pickler.dump(self, obj)
File "stringsource", line 2, in grpc._cython.cygrpc.Channel.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/tornado/ioloop.py:227: DeprecationWarning: clear_current is deprecated
IOLoop.clear_current()
/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first
self.make_current()
/home/ubuntu/venv/ERCOT_LMP/lib/python3.8/site-packages/distributed/nanny.py:860: DeprecationWarning: make_current is deprecated; start the event loop first
loop.make_current()
I think the important part of all this is File "stringsource", line 2, in grpc._cython.cygrpc.Channel.__reduce_cython__ TypeError: no default __reduce__ due to non-trivial __cinit__ but the function is returning a list of floats, so I don't understand the issue. Thanks for any help.
