2

What is an elagant/idiomatic way to achieve something like tuple unpacking with futures?

I have code like

a, b, c = f(x)
y = g(a, b)
z = h(y, c)

and I would like to convert it to use futures. Ideally I would like to write something like

a, b, c = ex.submit(f, x)
y = ex.submit(g, a, b)
z = ex.submit(h, y, c)

The first line of that throws

TypeError: 'Future' object is not iterable

though. How can I get a,b,c without having to make 3 additional ex.submit calls? ie. I would like to avoid having to write this as:

import operator as op
fut = ex.submit(f, x)
a = client.submit(op.getitem, fut, 0)
b = client.submit(op.getitem, fut, i)
c = client.submit(op.getitem, fut, 2)
y = ex.submit(g, a, b)
z = ex.submit(h, y, c)

I guess a potential solution is to write an unpack function like below,

import operator as op
def unpack(fut, n):
    return [client.submit(op.getitem, fut, i) for i in range(n)]

a, b, c = unpack(ex.submit(f, x), 3)
y = ex.submit(g, a, b)
z = ex.submit(h, y, c)

which works: for example if you first define:

def f(x):
    return range(x, x+3)
x = 5
g = op.add
h = op.mul

then you get

z.result() #===> 77

I thought something like this might already exist.

The above only works with dask.distributed.Future. It does not work for plain concurrent.futures.Future.

Daniel Mahler
  • 7,653
  • 5
  • 51
  • 90
  • 1
    "Throws an error." WHAT ERROR? – John Zwinck Sep 21 '17 at 03:00
  • @JohnZwinck `TypeError: 'Future' object is not iterable` – Daniel Mahler Sep 21 '17 at 03:59
  • `operator.itemgetter` can fetch multiple items. But look at its code; it's a class that does an iteration. – hpaulj Sep 21 '17 at 05:49
  • @hpaulj Thatnks! Iteration over the indexes is ok, it is iteration over the future that was the original problem. The above approach iterates over the indexes but the extacttion of the components is delayed through the `submit` call so it is ok. – Daniel Mahler Sep 21 '17 at 06:23
  • @hpaulj ah, but `itemgetter` will not work here because if I delay the cal to it l I will still end up with a future for an iterable, but I need an iterable of futures. The `unpack` usiing `getitem` is turning a future for an iterable into an iterable of futures for the elements, which is what make the unpacking assignment work. – Daniel Mahler Sep 21 '17 at 06:29

1 Answers1

3

A quick glance at:

https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future

suggests that you'll have to do something like

afuture = ex.submit(f, x)
a,b,c = afuture.result()
...

submit returns a Future object, not the result of running f(x).

This SO answer indicates that chaining futures is not trivial:

How to chain futures in a non-blocking manner? That is, how to use one future as an input in another future without blocking?

hpaulj
  • 221,503
  • 14
  • 230
  • 353
  • The `afuture.result()` call will block an the call to `f`, which defeats the purpose of using futures. – Daniel Mahler Sep 21 '17 at 03:55
  • 1
    @DanielMahler, can you give us an example of the chaining without the unpacking? Sounds like you've done some basic stuff with `futures`. Enlighten us. What are these 3 additional calls? – hpaulj Sep 21 '17 at 03:59
  • @hpaulj The desirable property is that nodes of the execution graph do not block on completion on nodes that they don't (transitively) depend on. This solution does not have this. The execution is sequential and the Future is useless. The code is equivalent to `a,b,c = fx)` – Ark-kun Nov 14 '22 at 00:53