74

My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks?

Does that mean we should use Redis as a message broker IF we are already using it for something else?

Pig
  • 2,002
  • 5
  • 26
  • 42
  • Celery clearly recommends using AMQP over Redis. I wouldn't use Redis. – DevLounge Apr 06 '17 at 20:56
  • 6
    @Apero Though Rabbitmq has been supported longer than Redis (and is the default), both are listed as stable. I don't see a clear recommendation either way. I'd be curious to read about if you've seen otherwise, however. http://docs.celeryproject.org/en/master/getting-started/brokers/index.html – Joe J Jun 18 '17 at 00:37
  • @DanilaGanchar the article mentioned: It is apparent that RabbitMQ takes 75% of Redis’ time to add a message and 86% of the time to process a message. why Redis is faster? – Tom Chen Jan 12 '21 at 03:03
  • Updated link to the brokers in the docs: https://docs.celeryproject.org/en/stable/getting-started/first-steps-with-celery.html#choosing-a-broker – Rmatt Mar 03 '21 at 16:11

2 Answers2

90

I've used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup.

  • If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise...
  • If you are using Redis for any other reason, go with Redis
  • If you are hosting at AWS, go with Redis so that you can use a managed Redis as service
  • If you hate complicated installs, go with Redis
  • If you already have RabbitMQ installed, stay with RabbitMQ

In the past, I would have recommended RabbitMQ because it was more stable and easier to setup with Celery than Redis, but I don't believe that's true any more.

Update 2019

AWS now has a managed service that is equivalent to RabbitMQ called Amazon MQ, which could reduce the headache of running this as a service in production. Please comment below if you have any experience with this and celery.

Mark Chackerian
  • 21,866
  • 6
  • 108
  • 99
  • 5
    One annoyance with using Redis / python 3.7 / Celery 4.2 is that the results backend doesn't work because `async` is now part of python ( see https://github.com/celery/celery/issues/4849 ) -- this should be fixed with celery 4.3 . I don't know if this also affects RabbitMQ. – Mark Chackerian Nov 16 '18 at 17:51
  • 14
    However, Celery 4.3 is now released, so this is no longer an issue -- results backend works again with Redis. – Mark Chackerian Apr 07 '19 at 16:15
  • 1
    You can also use Celery with Amazon SQS which is super simple to deploy and run – JasonGenX May 22 '19 at 15:31
  • @JasonGenX SQS support is experimental, though. I wonder what the unstable/missing bits are... – Tuukka Mustonen May 24 '19 at 08:00
  • 2
    As of July 2019, SQS is marked as stable. Yet, it is missing monitoring and control channels. Also, SQS behavior is a bit of counter-intuitive. We've got messages processed up to 14 times due to at-least-once delivery policy. – Yossi Jul 21 '19 at 08:46
  • According to this ticket https://github.com/celery/celery/issues/5001 you can not use Amazon MQ as replacement for RabbitMQ. They implement different verions of AMQP protocol (1.0 vs 0.9.1) – Igor Apr 23 '20 at 23:38
  • 2
    Now Amazon MQ also offers rabbitmq brokers in addition to activemq, I've used it and it works fine. – Ozgur Akcali Dec 08 '20 at 10:04
  • I've personally moved away from SQS so I don't know if this works/is appropriate for Celery, but for those reading along: SQS now supports FIFO queues that guarantee FIFO and exactly-once delivery. – Kye Sep 29 '22 at 03:17
3

The Redis broker gives tasks to workers in a fair round robin between different queues. Rabbit is FIFO always. For me, a fair round robin was preferable and I tried both. Rabbit seems a tad more stable though.

Math is Hard
  • 896
  • 1
  • 12
  • 24
  • "The Redis broker gives tasks to workers in a fair round robin between different queues." Can you provide a reference for this? – mrpandey Feb 14 '23 at 09:59
  • 1
    https://docs.celeryq.dev/projects/kombu/en/v4.0.2/reference/kombu.transport.redis.html Believe me - I tried all the different transport options. I'm actually using SQS now and to me, it's the best one but it's also not round robin. – Math is Hard Feb 14 '23 at 21:08