4

I am abusing dask as a task scheduler for long running tasks with map(, pure=False). So I am not interested in the dask graph, I just use dark as a way to distribute unix commands.

Lets say if have 1000 tasks and they run for a week on a cluster of 30 workers. What I have seen is that if a worker goes down, its tasks get redistributed to the remaining workers.

Sometimes I can free resources from other simulations and I add new workers to the desk cluster. However, those workers then have 0 tasks assigned and idle. They only get new tasks if one of the old workers goes down, then the tasks get redistributed.

So, my question: "Can I manually redistribute and shuffle the tasks on the dask cluster"?

MaxBenChrist
  • 547
  • 3
  • 9

1 Answers1

1

The scheduler should already be balancing old tasks to new workers. Information on our work-stealing policies live here:

http://distributed.readthedocs.io/en/latest/work-stealing.html

So I'm surprirsed to hear that tasks don't automatically rebalance. If you're able to produce an mcve to reproduce the problem I'd enjoy taking a look.

MRocklin
  • 55,641
  • 23
  • 163
  • 235