amqp exchange types

Question

I'm getting into the whole amqp thing and i have a question regarding which type of exchange type to use under the following scenario's:

1) i have the need to create a worker pool where each worker does something when they receive a message. now i want different workers attached to different types of tasks; which i can specify by using the routing keys of each message in a topic fashion. on the consumer end, playing around a bit with kombu i notice that if i specify the same queue name but with different routing keys i can not 'filter' the messages. eg if i have one consumer with '#' and another with 'foo.#' - both using the same queue name, the latter consumer will work round robin on the queue with the former consumer. is this expected? i am running both consumers on the same machine.

2) so given that, i construct unique queue names for each consumer and this time, each consumer does only get what i ask for with the routing key. however, because they are distinct queues, i may get a task in more than just one consumer. eg if consumer 1 has key '#', and consumer 2 has 'foo.#'; when consumer 2 receives (and acks) a message, consumer 1 also gets the same message. this is not what i want; i would like only one consumer to get the message only. is there a way i can achieve this without writing a 'task manager'?

cheers,

Answer 1

For most people it is best to just use a topic exchange for everything until you fully understand how AMQP works. You can get fanout and direct behavior just by choosing the right binding key for a queue. For instance if you use "#" for a binding key, then that queue behaves as if it was connected to a direct exchange. And if you bind two or more queues to the same routing keys, then those queues function as it if was a fanout exchange.

The round robin behavior is expected. Both tasks are subscribed to the exact same queue. The fact that the binding keys are different just confuses everything. Probably whoever binds last, will set the binding key for every queue user. Best not to do that. I build a system in which several queues have anywhere from 4 to 15 instances of the exact same worker code, pulling messages off the same queue and then collecting data from web services. I have even had the workers running on different CPUs although in the end that was not necessary for performance.

I'm not sure why you are using wildcards in the binding keys. If you have 8 consumers named A through H, and each one does a different job, then why not publish messages with routing keys work.A through work.H and then use the same binding keys work.A through work.H. That way if you have multiple instances of worker B, they all bind to work.B and no message is delivered twice.

Also, if you don't ack a message after handling it, then eventually it will go back on the queue and be delivered again. Hopefully you are acking after successfully handling the message. No task manager is needed, just better understanding of all the AMQP knobs.