6

I want to apply the Client.map method to a function that uses multiple arguments as does the Pool.starmap method of multiprocessing. Here is an example

from contextlib import contextmanager

from dask.distributed import Client


@contextmanager
def dask_client(**kwargs):
    """some docs"""
    kwargs.setdefault("ip", "localhost:8786")
    client = Client(**kwargs)

    try:
        yield client
    except Exception:
        raise
    finally:
        client.close()


def f(x,y,z):
    return x+y+z
# Dummy function

if __name__ == "__main__":
    with dask_client() as client:
        client.map(f, (1,2,3), (1,2,3))

distributed.worker - WARNING - Compute Failed
Function:  f
args:      (1, 1)
kwargs:    {}
Exception: TypeError("f() missing 1 required positional argument: 'z'")

distributed.worker - WARNING - Compute Failed
Function:  f
args:      (2, 2)
kwargs:    {}
Exception: TypeError("f() missing 1 required positional argument: 'z'")

distributed.worker - WARNING - Compute Failed
Function:  f
args:      (3, 3)
kwargs:    {}
Exception: TypeError("f() missing 1 required positional argument: 'z'")

This was the accepted answer here

I know that each tuple is being viewed as x of my function f. If possible I don't want a solution like

def f(var_list):
    # could be sum(), but this is a dummy example
    return var_list[0] + var_list[1] + var_list[2]
SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46
Andrex
  • 602
  • 1
  • 7
  • 22

1 Answers1

4

You're close, note that there should be the same number of iterables as the arguments in your function:

from dask.distributed import Client
client = Client()

def f(x,y,z):
    return x+y+z

futs = client.map(f, *[(1,2,3), (4,5,6), (7,8,9)])

client.gather(futs) # [12, 15, 18]

From the comments it seems you want to store all arguments in a single tuple, in that case you can do:

# will pass them as x=1, y=2, z=3
long_list = [(1,2,3), (4,5,6), (7,8,9), (10,11,12)]

futs = client.map(f, *zip(*long_list))

client.gather(futs) # [6, 15, 24, 33]
SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46