4

Am building a custom graph for one operation with Dask. Am familiar with how to pass arguments to a function in Dask graph and have read up on the docs. However still seem to be missing something.

One of the functions used in the Dask graph takes keyword arguments. Though am confused as to how the keyword arguments can be passed to it. Some of these keyword arguments represent Dask objects so they must be in the graph explicitly (i.e. functools.partial won't work). Can see the following options.

  1. Try to pass any keyword arguments in the correct positional order. (won't work for generic **kwargs or keyword only arguments)
  2. Write a wrapper that converts positional arguments to keywords arguments and use that in the graph. (works ok, but has some overhead)
  3. Include them directly in the graph somehow? (ideal, but unclear how)

Any thoughts on how best to do this?

jakirkham
  • 685
  • 5
  • 18

2 Answers2

1

A common workaround is to use the apply function

from dask.utils import apply
task = (apply, func, args, kwargs)  # func(*args, **kwargs)
jakirkham
  • 685
  • 5
  • 18
MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • Does this imply that the following should work? ``def func(a, b, c=None): return a*b*c; dsk['d'] = (apply, func, ['a', 'b'], {'c': 'c'})`` where ``'a', 'b', 'c'`` are all keys to well-defined tasks. I cannot make this work -- the string ``'c'`` is always passed to func. Not clear to me if this is not supported or if it's a bug. Thanks in advance. – Will Holmgren Jul 04 '18 at 00:08
  • I recommend making a new question or issue so that people who follow the dask tag will see your question. As it is now, only I get informed. – MRocklin Jul 04 '18 at 01:35
0

The answer by @MRocklin should be edited to conform to newer dask versions:

from dask.utils import apply
task = (apply, func, args, kwargs)  # func(*args, **kwargs)
Martin Raspaud
  • 76
  • 2