dask future not updating according to progress

Question

problem...

1. I am submitting function to dask client and recording the futures keys.

2. I am using these key to instantiate the futures in a different fucntion.

3. These futures are stacked in "pending" mode

backbone of the program I am trying:

from dask.distributed import Client, Future
client = Client()

def func1(client, data):
    future = client.submit(some_long_function, data)
    key = future.key
    return key

key = func1(client, data)

def func2(key):
    future = Future(key, client)
    print(future)  # !! Always show "pending" even when the process finished !

How can I use the "key" of a future to get the status of the run and retrieve the results when done ?

(NOTE: I don't want to use the "Future" object itself, since I would like to send this "key" to a client javascript application)

Answer 1

When the future gets garbage collected in func1 the client sends a message to the scheduler that it no longer needs the result and so it probably never runs. You probably want to call fire_and_forget on the future to ensure that it runs, even if you don't hold a reference to the future.

See these docs:

• Futures
• fire_and_forget