0

Due to my experience with Python I am currently in need to pick one of task queues solutions available for that language. I am considering Celery (I should mention that I do not use Django and switching to that framework is not an option), Python-RQ(http://python-rq.org/) and Zeromq with pyzmq.

I would be glad if someone could share his/her experiences with those or other task queues in production environment. Right not I am considering strongly python-rq.

Regards

loleknwn
  • 115
  • 7
  • 4
    "Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – zero323 Sep 20 '13 at 12:26

1 Answers1

3

I chose celery with Redis (and no django), and am currently happy with it. Things I ran into:

  • RabbitMQ works until it doesn't... it started pausing for no reason, and then eventually hanging all tasks. Since it was heavyweight, and chosen for reliability, it got tossed overboard.

  • Zeromq looked like a fabulous project, and it seemed to have great performance, but when I tried to use it, it failed to pass any messages, even using the built in examples. The fault was certainly mine -- ipython also uses zeromq and works flawlessly on the same machine.

  • The actual work required to get celery up and running was minimal, for me. I has about the same usecase as python-rq seems to address, but it didn't exist when I started, so I used celery, and it (apart from RabbitMQ, which was a pain for me), it was easy to set up and use.

  • Configuring Redis to not use any persistent storage made things easier to predict, and I never needed to worry about queue state.

Things I didn't run into/use:

  • My tasks are not persistent or transactional. If a task dies, the user gets an error message, and can try again. The tasks are also idempotent -- if they are executed muliple times nothing goes wrong. If that's not true for your case, you'll need to be more careful about how you handle tasks, and rabbitmq will be more attractive.

  • my tasks did not conflict or need to execute in a given order.

  • my tasks were chosen to be fairly coarse grained -- 9.2 seconds and up.

Henry Crutcher
  • 2,137
  • 20
  • 28
  • I built projects in zmq and they worked flawlessly. The examples can be confusing at first. I've also built projects with Celery tasks and they work really well, until they don't. They use up a ton of memory depending on the number of tasks. – Blairg23 Dec 01 '15 at 23:22
  • I'm curious which example you got to work -- when I tried running the examples, they didn't confuse me. They simply failed to send any messages. I'm sure it was something I did, but there was no documentation as to why that might happen, or anywhere I could ask for help. – Henry Crutcher Dec 05 '15 at 20:44
  • I got the Python example to work. In fact, I found a fallacy in it and fixed it and sent in the fix to the ZMQ people so it should be a working example now. – Blairg23 Dec 06 '15 at 01:52
  • I should elaborate. I got the Python PubSub working – Blairg23 Dec 06 '15 at 23:18