52

There seem to be different implementations of task/job queues for Python 3:

  1. Celery, popular but apparently unmaintained and stale;
  2. RQ, of which I have little information;
  3. TaskTiger, similarly to RQ I know little about it;
  4. Huey , similarly to RQ I know little about it;
  5. WorQ had its last update in 2016.

Then there are “cloud” based solutions like Google’s Task Queue API or AWS’s Cloud Watch Events, but that’s more of a last resort.

For my project I am looking for a stable and active task queue implementation. I’ve used Celery for the past year, but the lack of support and non-attention to existing bugs is worrisome.

What alternatives exist?

NotLawson
  • 3
  • 1
Jens
  • 8,423
  • 9
  • 58
  • 78
  • 4
    Related writeup with more links: https://www.fullstackpython.com/task-queues.html And here is a collection of queuing frameworks: http://queues.io/ – Jens Nov 20 '17 at 10:58
  • 24
    Celery is neither unmaintained nor stale. Issues are being closed. It is still the de-facto standard in the Python industry. Not saying is the only valid alternative, just saying it is maintained and robust. – Akhorus Jan 16 '18 at 20:44
  • 3
    @Akhorus: AFAIK left the maintainer of Celery the project, and it has been sitting around and accumulating issues without support (ICQ, Github, mailing lists) for months. There’s been a flurry of activity over the past few weeks, and it looks like somebody else has taken over the maintenance. – Jens Jan 17 '18 at 03:46
  • That is not true... Just look at how many issues have been closed, and they are about to make a stable release 4.2.0 (it is in the RC3 state now). – DejanLekic May 10 '18 at 08:21
  • Development for windows stopped at Celery 3 so you could say its "stale" if thats what you meant – Clint Jul 24 '18 at 22:10

3 Answers3

76

I wrote Dramatiq specifically because of my increasing frustration with Celery having used it professionally for years. Check out the motivation page the "why" and a feature comparison between Dramatiq, Celery and RQ. Some highlights:

  • actively developed and used in production
  • great docs
  • automatic retries
  • code auto-reload
  • locks and rate limiting
  • redis and rabbitmq support
Bogdan Popa
  • 1,081
  • 7
  • 6
  • 2
    I’ve been using Dramatiq for [Bookalope](https://bookalope.net/) now for about a month, and it works like a charm. Great work, Bogdan – Jens Jul 12 '18 at 23:49
  • 2
    Dramatiq is really good - simple, flexible, powerfull – Vladimir Nov 06 '19 at 05:46
  • 2
    Used Dramatiq in a few projects and definitely recommend it – Andrii Rusanov Nov 15 '19 at 11:26
  • 1
    Task Prioritization, auto-retry are now supported in Celery - https://docs.celeryproject.org/en/stable/faq.html#does-celery-support-task-priorities – Gh0sT Feb 13 '20 at 14:26
  • Hey @Bogdan Pop can you tell me how can i run it as a daemon in linux based environment ? Or anyother suitable way to deploy it in production ? – ruhaib Mar 02 '20 at 18:47
  • @BogdanPopa Does it support Windows OS, because currently I am using celery for windows and I am having trouble with ConnectionResetError[10054]. – sattva_venu Sep 13 '20 at 05:18
  • @sattva_venu yes, Dramatiq does support Windows. – Bogdan Popa Sep 14 '20 at 07:39
  • @BogdanPopa how about remote system configuration, can we mention another host's IP address and trigger the worker in different machine using Dramatiq? – sattva_venu Sep 14 '20 at 17:08
  • Really want dramatiq to work. I'm frustrated as hell with celery, it isn't reliable at all to run in production. – AJ. Oct 01 '20 at 07:54
  • I'm looking for celery alternatives, but revoking a task is one of our functionality which i couldn't see in dramatiq. – Satyaaditya Sep 28 '21 at 08:17
  • Any guidance for the error `dramatiq_abort missing key word argument "client"` as from your cookbook page https://dramatiq.io/cookbook.html on the abort section... – Shmack Oct 15 '21 at 18:15
13

One new alternative is django-carrot, which came about due to our requirement for a lightweight alternative to Celery, while still providing support for RabbitMQ (I think the other alternatives you've mentioned are all based on Redis)

Django-carrot is still under development but is expected to go into production at my company in the next few weeks

Disclaimer - I'm the author of django-carrot

Christopher Davies
  • 174
  • 7
  • Thanks! How’s that coming along, and would it run standalone (i.e. without django) as well, or with [Pyramid](https://github.com/Pylons/pyramid)? – Jens Nov 20 '17 at 09:09
  • We've been using it live since the middle of October. Its designed to work with Django only (as this is what all our web apps use). Because of the number of long running tasks running through our Django apps, which are very sensitive to end user inputs, a key part of our use case is to be able to write the logs to the project DB so that they can be tracked and debugged easily – Christopher Davies Nov 27 '17 at 09:19
1

RabbitMQ guys recommend Pika library: https://github.com/pika/pika You can find simple and intuitive examples on their website: https://www.rabbitmq.com/tutorials/tutorial-one-python.html

gSorry
  • 1,254
  • 2
  • 21
  • 29
  • 4
    For anyone that stumbles on this: We tried to roll our own job queue at $DAYJOB from pika and I am now exploring libraries to replace that system. – AstraLuma Mar 02 '21 at 18:34
  • 2
    Pika is not a task queue by any means - it is client library, which is a lot simpler (and can be used to implement custom task queue). – Eugene Feb 17 '22 at 09:40