Dask: handling unresponsive workers

Question

When using Dask with SGE or PBS clusters I sometimes have workers becoming unresponsive.
These workers are highlighted in red in the dashboard Info section with their "Last seen" number constantly increasing.

I know this can happen if submitted tasks hold the GIL for too long but that's not the case here. I'm talking about workers for which something went wrong (probably unrelated to dask or the task itself).
They will not come back and are not detected as dead either.

The problem is that tasks submitted on these workers (they become unresponsive after receiving a task, maybe when loading the environment) never end and block everything.

Is there a setting allowing to "timeout" or "invalidate" a worker if it was unresponsive for a given time ?

If not, is it possible and what would be the recommended way to manually do this invalidation and dispatch remaining tasks on other workers ?

Thanks in advance for any help regarding this issue.

Answer 1

If anyone is wondering the same thing, the answer was given in the following topic.

Setting the worker-ttl parameter allows to consider a worker as dead if not seen for a specified amount of time.

   worker-ttl:
    type:
    - string
    - "null"
    description: |
      Time to live for workers.
      If we don't receive a heartbeat faster than this then
      we assume that the worker has died.

This can be set using dask

dask.config.set({"distributed.scheduler.worker-ttl": "120min"})