Nested dask delayed or futures

Question

Looking for best practice for nested parallel jobs. I couldn't nest dask delayed or futures so I mixed both to get it to work. Is this not recommended? Is there better way to do this? Example:

import dask
from dask.distributed import Client
import random
import time
client = Client()

def rndSeries(x):
    time.sleep(1)
    return random.sample(range(1, 50), x)

def sqNum(x):
    time.sleep(1)
    return x**2

def subProcess(li):
    results=[]
    for i in li:
        r = dask.delayed(sqNum)(i)
        results.append(r)
    return dask.compute(sum(results))[0]

futures=[]
for i in range(10):
    x = client.submit(rndSeries,random.randrange(5,10,1))
    y = client.submit(subProcess, x)
    futures.append(y)
client.gather(futures)

SultanOrazbayev · Answer 1 · 2021-02-05T16:58:22.930

Consider modification of your script to have a deterministic workflow. If you start with 1 worker, you will see that the process completes in 20 seconds (as expected, 2 processes of 1 second + 6 processes of 3 seconds). If you have 2 workers, the execution time will drop to 10 seconds.

import dask
from dask.distributed import Client, LocalCluster
import time
import numpy as np

cluster = LocalCluster(n_workers=1, threads_per_worker=1)
client = Client(cluster)

# if inside jupyter split the code below into a new cell
# to see accurate timing
%%time
def rndSeries(x):
    time.sleep(1)
    return np.random.rand()

def sqNum(x):
    time.sleep(3)
    return 1

def subProcess(li):
    results=[]
    li = [1,2,3]
    for i in li:
        r = dask.delayed(sqNum)(i)
        results.append(r)
    return dask.compute(sum(results))[0]

futures=[]
for i in range(2):
    x = client.submit(rndSeries, np.random.rand())
    y = client.submit(subProcess, x)
    futures.append(y)
client.gather(futures)

What happens if you have 6 workers? Execution time is now 4 seconds (the lowest possible for this task), so it seems that the only drawback of dask.compute() inside a future version is that it forces the results of delayeds to be on a single worker. This is probably OK in many cases, however, if the combined resource requirements of all delayed tasks exceed resources of a single worker, then the best way to proceed is to submit tasks from tasks: https://distributed.dask.org/en/latest/task-launch.html

Interesting. Limiting the number of concurrent delayed jobs to number of future thread will be a problem for me. I need to run 1000 futures and about 10000 delayed tasks for each future. I'll take a look at tasks. — J.Sung, Feb 05 '21 at 23:56

Nested dask delayed or futures

1 Answers1