0

I have a list of independent tasks and each needs different resources and takes different calculation times. I have to specify resource constraints on each of these tasks in the list and set the priority for the task with the least amount of resources to be finished first.

@dask.delayed()
def func(...)
...
 
task_list = []
...
   task_list.append(dask.delayed(func)(...))

dask.compute(*task_list, resources={'memory':10e9}, optmize_graph=False)

In the above code, I could assign a resource constraint on all the tasks in the list but instead, i would like to assign different resource constraints for each of the tasks in the list.

something like this:

task_list = [delayed(func1..), delayed(func2) .....]

delayed(func1) - {'memory':10e9} # this task needs 10 GB of memory to start
delayed(func2): {'memory':5e9} # # this task needs 5 GB of memory to start
...

dask.compute(*task_list)

Could someone please help.

Thanks !

ranjith
  • 115
  • 2
  • 10

1 Answers1

1

As of 2020-08-07 Dask does not support this. It is a frequently requested feature though. You may want to go through the github issues to see if you can find similar people and share your thoughts.

MRocklin
  • 55,641
  • 23
  • 163
  • 235