1

The documentation for the Django ORM transport for Celery says that using more than a few workers can cause some tasks to be executed more than once.

http://docs.celeryproject.org/en/latest/getting-started/brokers/django.html

Can anyone quantify what "more than a few" means? Concretely speaking, I have two servers and would like to run two celery workers. I am wondering if anyone can tell me if having two workers exposes me to the risk that some tasks will execute more than once.

Krystian Cybulski
  • 10,789
  • 12
  • 67
  • 98
  • thats very likely to happen, for that reason you should write your tasks as idempotent functions so that multiple execution have no side effects (other than wasting computing resources). – Tommaso Barbugli Sep 15 '13 at 20:35
  • btw in my experience using django orm as a broker is not a great idea. in general every broken different than rabbitmq always gave lots of trouble. rabbitmq is pita to setup but works like a charm once you got it running properly :) – Tommaso Barbugli Sep 15 '13 at 20:38
  • writing idempotent functions sounds like a good idea, but I have not figured out how to write email sending in such a fashion. :) We are presently planning on running with one worker as the workload is low enough to allow that. In the future, we are surely going to RabbitMQ, but I am trying to plan ahead. – Krystian Cybulski Sep 17 '13 at 07:24
  • i guess you can assign every email an unique identifier (eg. subject / recipient) and store that somewhere (easy to check if you sent that message less than x seconds ago) – Tommaso Barbugli Sep 17 '13 at 14:02

0 Answers0