How to resubmit dask jobs (from previous dead workers) when new workers are added after old workers have died?

Question

Is there a way in dask to resubmit jobs (from previous dead workers) when new workers are added after old workers have died?

Might it be possible to achieve this using a scheduler plugin distributed.diagnostics.plugin.SchedulerPlugin?

Answer 1

If you are using future API, then you can examine the status property of the individual future objects using .status and take actions accordingly. For example, if f is a future object then we can ask the scheduler to retry its calculation based on some condition like this:

if f.status.isin(['error', 'cancelled', 'lost']):
   f.retry()