3

Below code uses appx 1 sec to execute on an 8-CPU system. How to manually configure number of CPUs used by dask.compute eg to 4 CPUs so the below code will use appx 2 sec to execute even on an 8-CPU system?

import dask
from time import sleep

def f(x):
    sleep(1)
    return x**2

objs = [dask.delayed(f)(x) for x in range(8)]
print(dask.compute(*objs))  # (0, 1, 4, 9, 16, 25, 36, 49)
SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46
Russell Burdt
  • 2,391
  • 2
  • 19
  • 30

1 Answers1

3

There are a few options:

  1. specify number of workers at the time of cluster creation
from dask.distributed import Client

# without specifying unique thread, the function is executed
# on all threads
client = Client(n_workers=4, threads_per_worker=1)

# the rest of your code is not changed
  1. specify how many (and which) workers should execute a task

client = Client(n_workers=8, threads_per_worker=1)

list_workers = list(client.scheduler_info()['workers'])

client.compute(objs, workers=list_workers[:4]) 

# submit only to the first 4 workers
# note that workers should still be single-threaded, but the difference
# from option 1 is that you could in principle have more workers
# that are idle, also the `workers` kwarg can be passed to
# dask.compute rather than client.compute
  1. specify a semaphore
from dask.distributed import Client, Semaphore

client = Client()
sem = Semaphore(max_leases=4, name="foo")

def fmodified(x, sem):
    with sem:
        return f(x)

objs = [dask.delayed(fmodified)(x, sem) for x in range(8)]
print(dask.compute(*objs))  # (0, 1, 4, 9, 16, 25, 36, 49)

Update: as noted by @mdurant in the comments, if you are running this in a script, then if __name__ == "main": is needed to guard the relevant code from being executed by workers. For example, the second option from the list above would look like this in a script:

#!/usr/bin/env python3
import dask
from dask.distributed import Client
from time import sleep

def f(x):
    sleep(1)
    return x**2

objs = [dask.delayed(f)(x) for x in range(8)]

if __name__ == "main":
    client = Client(n_workers=8, threads_per_worker=1)

    list_workers = list(client.scheduler_info()['workers'])

    results = client.compute(objs, workers=list_workers[:4])

    print(results)
SultanOrazbayev
  • 14,900
  • 3
  • 16
  • 46
  • 1
    Option (1) still is using all 8 CPUs (appx 1 sec to execute). Thinking I needed to change ```dask.compute(*objs)``` to ```client.compute(*objs)``` but led to ```TypeError: Truth of Delayed objects is not supported```. Option (2) needed a ```client``` object which was created off of Option (1) but then led to ```TypeError: compute() got multiple values for argument 'workers'```. Option (3) works but not preferred due to the additional function. Any modification to (1) or (2) to get those running? – Russell Burdt Sep 30 '21 at 18:00
  • Also experienced an intermittent ```RuntimeError``` with all three options: ```RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable.``` – Russell Burdt Sep 30 '21 at 18:02
  • Hmm, sorry about this... let me check.. – SultanOrazbayev Sep 30 '21 at 18:18
  • I did not notice the single-threading behaviour before... so that's interesting. Re: `freeze_support` messages, I have never seen that happen... so might be specific to your code/setup (maybe you are running on Windows) – SultanOrazbayev Sep 30 '21 at 18:32
  • Yes, running on Windows. Never saw that message on same system in all previous use of ```dask``` importing only ```delayed``` and ```compute``` objects from ```dask```. It was ```from dask.distributed import Client, LocalCluster``` and using those objects that intermittently led to the ```freeze_support``` messages... – Russell Burdt Sep 30 '21 at 18:37
  • Ok, can get Option (1) to run now but it is very touchy as soon as the ```Client``` object is instantiated. For example, will not run directly without the ```freeze_support``` issue. Will run if I copy-paste line-by-line in a console, but then the ```dask.diagnostics.ProgressBar``` no longer works. This is all on Windows, maybe the functionality is better on Linux... otherwise ```dask``` has worked well on Windows and Linux if the ```Client``` object is not imported and used. – Russell Burdt Sep 30 '21 at 19:03
  • 2
    The freeze thing is when you save this code into a file, as opposed to running in an interactive session or notebook. In a file, you need an `if __name__ == "main":` block, else each subprocess imports the file and tries to launch more workers. – mdurant Feb 22 '22 at 14:13