Jobs firing to wrong celery

Question

I am using django celery and rabbitmq as my broker (guest rabbit user has full access on local machine). I have a bunch of projects all in their own virtualenv but recently needed celery on 2 of them. I have one instance of rabbitmq running

(project1_env)python manage.py celery worker
 normal celery stuff....
[Configuration]
   broker:      amqp://guest@localhost:5672//
   app:         default:0x101bd2250 (djcelery.loaders.DjangoLoader)

[Queues]
   push_queue:  exchange:push_queue(direct) binding:push_queue

In my other project

(project2_env)python manage.py celery worker
 normal celery stuff....
[Configuration]
   broker:      amqp://guest@localhost:5672//
   app:         default:0x101dbf450 (djcelery.loaders.DjangoLoader)

[Queues]
   job_queue:   exchange:job_queue(direct) binding:job_queue

When I run a task in project1 code it fires to the project1 celery just fine in the push_queue. The problem is when I am working in project2 any task tries to fire in the project1 celery even if celery isn't running on project1.

If I fire back up project1_env and start celery I get

Received unregistered task of type 'update-jobs'.

If I run list_queues in rabbit, it shows all the queues

...
push_queue  0
job_queue   0
...

My env settings and CELERYD_CHDIR and CELERY_CONFIG_MODULE are both blank.

Some things I have tried:

purging celery
force_reset on rabbitmq
rabbitmq virtual hosts as outlined in this answer: Multi Celery projects with same RabbitMQ broker backend process
moving django celery setting out and setting CELERY_CONFIG_MODULE to the proper settings
setting the CELERYD_CHDIR in both projects to the proper directory

None of these thing have stopped project2 tasks trying to work in project1 celery.

I am on Mac, if that makes a difference or helps.

UPDATE

Setting up different virtual hosts made it all work. I just had it configured wrong.

score 3 · Accepted Answer · answered Feb 01 '13 at 20:29

3

If you're going to be using the same RabbitMQ instance for both Celery instances, you'll want to use virtual hosts. This is what we use and it works. You mention that you've tried it, but your broker URLs are both amqp://guest@localhost:5672//, with no virtual host specified. If both Celery instances are connected to the same host and virtual host, they will produce to and consume from the same set of queues.

answered Feb 01 '13 at 20:29

Loren Abrams

1,002
8
9

I have tried it `BROKER_URL = 'amqp://guest@localhost/project2'` and when I run celery worker I even get the correct broker. But my tasks are still falling in to project1 celery. I will take another look at this solution. I might have set something up wrong. – Raisins Feb 01 '13 at 20:33
Wanted to say I got back to this today. Your solution worked. It was 2 problems. 1# I did have vhosts setup wrong. 2# I accidently added the queue to my apply_async call and had the wrong name there, so even if they got sent they where going to the wrong place. Got the 2 projects confused. – Raisins Feb 05 '13 at 22:09

Jobs firing to wrong celery

1 Answers1