3

I have a function interpolate_to_particles written in c and wrapped with ctypes. I want to use dask.delayed to make a series of calls to this function.

The code runs successfully without dask

# Interpolate w/o dask
result = interpolate_to_particles(arg1, arg2, arg3)

and with the distributed schedular in single-threaded mode

# Interpolate w/ dask
from dask.distributed import Client
client = Client()
result = dask.delayed(interpolate_to_particles)(arg1, arg2, arg3)
result_c = result.compute(scheduler='single-threaded')

but if I instead call 

result_c = result.compute()

I get the following KeyError:

> Traceback (most recent call last):   File
> "/path/to/lib/python3.6/site-packages/distributed/worker.py",
> line 3287, in dumps_function
>     result = cache_dumps[func]   File "/path/to/lib/python3.6/site-packages/distributed/utils.py",
> line 1518, in __getitem__
>     value = super().__getitem__(key)   File "/path/to/lib/python3.6/collections/__init__.py",
> line 991, in __getitem__
>     raise KeyError(key) KeyError: <function interpolate_to_particles at 0x1228ce510>

The worker logs accessed from the dask dashboard do not provide any information. Actually, I do not see any information that the workers have done anything besides starting up.

Any ideas on what could be occurring, or suggested tools that I can use to further debug? Thanks!

elltrain
  • 82
  • 4
  • Do you have a minimal example? https://stackoverflow.com/help/minimal-reproducible-example. Do you need to create the Client, or does the error surface with just the single-threaded scheduler? – TomAugspurger Apr 07 '20 at 14:02
  • @TomAugspurger Thanks for the comment. Interestingly, the minimal examples I have constructed (based on [post 1](https://stackoverflow.com/questions/60941300/can-i-use-dask-delayed-on-a-function-wrapped-with-ctypes?rq=1) and [post 2](https://stackoverflow.com/questions/61000054/dask-delayed-function-call-with-non-passed-parameters)) do not suffer the same issue. The error occurs only if I create the client. That is, the call `result = dask.delayed(interpolate_to_particles)(...)` followed by `result_c = result.compute()` works as intended. – elltrain Apr 07 '20 at 15:35

1 Answers1

0

Given your comments it sounds like your function does not serialize well. To test this, you might try pickling the function in one process, and try unpickling it in another.

>>> import pickle
>>> print(pickle.dumps(interpolate_to_particles))
b'some bytes printed out here'

And then in another process

>>> import pickle
>>> interpolate_to_particles = pickle.loads(b'the same bytes you had before')

If this doesn't work then you'll know that that's your problem. I would encourage you to look up "how to make sure that ctypes functions are serializable" or something similar, or ask another question with that smaller scope here on Stack Overflow.

MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • 1
    I suspect this is the issue. Below the KeyError noted above is the message `During handling of the above exception, another exception occurred:` `ValueError: ctypes objects containing pointers cannot be pickled`. I recently read [your post](http://matthewrocklin.com/blog/work/2013/12/05/Parallelism-and-Serialization), and I now understand that serializing is an essential to parallelism. I also now understand that [serializing objects with pointers](https://stackoverflow.com/questions/9768218/how-to-save-ctypes-objects-containing-pointers) is problematic. – elltrain Apr 14 '20 at 03:37
  • I have the same issue and pickling works fine, which is no surprise since all my arguments are ints, strings or dicts. The error appeared out of nowhere, it was working fine and now it doesnt – Nimitz14 Apr 27 '20 at 17:44
  • (slightly different setting, using `SGECluster` and `Client::submit` – Nimitz14 Apr 27 '20 at 17:45