How to assign tasks to specific worker within dask yarn cluster

Question

The analogue of this question for dask.distributed can be found here: How to assign tasks to specific worker within Dask.Distributed

But the same technique doesn't seem to work for the yarn cluster of dask. Is there a way to do it in yarn cluster dask case? I.e. assign a task to specific worker in a specific node with certain ip address?

Answer 1

I will answer my own question. In short, the post that I put the link DOES work. But you must make sure that the method you are assigning to the worker gets different arguments. Otherwise, it will be run on the same container.