3

I have a django 1.3 site using SQLite. It works well.

I recently install django-celery along with django-kombu. I am using supervisord to launch the celery dameon with the command:

python manage.py celeryd -v 2 -B -s celery -E

Everything works, except launching celeryd brings the system to a crawl. When I use top, I see that I am creating a long running python process that consumes anywhere from 5% to 10% of the cpu on my shared server.

During testing, I have no celery tasks scheduled or running.

Chuck
  • 1,089
  • 1
  • 13
  • 18
  • 1
    This site really needs a way to flag closings as abusive. This can not possibly be closed in good faith. Good grief. – Shayne Dec 02 '14 at 02:54

1 Answers1

0

From my experience, it's django-kombu that's taking a lot of CPU time. Use another broker like RabbitMQ or Redis.

Etienne
  • 12,440
  • 5
  • 44
  • 50
  • Closed? Are you sure? Every part of my question is directly related to software development. I honestly cannot see the difference between my question and the related ones on the right side of this page. – Chuck Dec 06 '11 at 07:04
  • Sorry, I'm not the one who closed your question. I really don't know why `casperOne` did that? It seems a valid question to me. Unfortunately it doesn't look you can reopen it: http://meta.stackexchange.com/questions/36415/how-do-you-reopen-a-closed-question – Etienne Dec 06 '11 at 15:29
  • But I can elaborate on my answer. When I tried django-kombu I had the same kind of results as you. I asked why on the celery IRC but I don't think I got any answer. I think one part of the problem is that kombu need to poll the DB very often (but I'm pretty sure it's not the only reason). If you can install/use RabbitMQ/Redis on your shared hosting, go that way. It's really simple and a lot more efficient. If your needs are simple my choice will be Redis, it's more simple to setup/manage and you can use it for Django cache, etc. – Etienne Dec 06 '11 at 15:38
  • Thanks Etienne. Your answer is very helpful. I noticed that most of the users that raved about Celery were using RabbitMQ. It's definitely worth a try. My web site is effectively dead until I solve this. I was holding off on commenting or accepting your answer until I could implement it and test it. It will be about a week for me to do that. I will report what I find. – Chuck Dec 06 '11 at 18:23
  • Update - I switched to Redis. Eveything is working well now. Although if I run the "top" command, it still says that my process is eating up a lot of cpu. I am on a shared server on webfaction. I seriously doubt that my process is allowed to use as much cpu as is being reported. Perhaps, the process that is using all that cpu has such a low priority that is does not interfere with anyone else or my django processes? – Chuck Dec 18 '11 at 23:25