0

I have been using celery for a while but am looking for an alternative due to the lack of windows support.

The top competitors seem to be dask and dramatiq. What I'm really looking for is for something that can distribute 1000 long running tasks onto 10 machines. Each should pick up the next job when it has completed the task, and give a callback with updates (in celery this can be nicely achieved with @task(bind=True), as the task instance itself can be accessed and I can send the status back to the instance that sent it with an update).

Is there a similar functionality available in dramatiq or dask? Any suggestions would be appreciated.

Gokhan Sari
  • 7,185
  • 5
  • 34
  • 32
Nickpick
  • 6,163
  • 16
  • 65
  • 116

1 Answers1

1

On the Dask side you're probably looking for the futures interface : https://docs.dask.org/en/latest/futures.html

Futures have a basic status like "finished" or "pending" or "error" that you can check any time. If you want more complex messages then you should look into Dask Queues, PubSub, or other intertask communication mechanisms, also available from that doc page.

MRocklin
  • 55,641
  • 23
  • 163
  • 235