4

Maybe I am asking the wrong question here.

what I'm trying to do: multiple producers push data in dynamic categories into a Named exchange. multiple consumers need to pick up this data from these dynamically named queues and act on them.

the problem is that all of the examples of consumption that I see require the consumer/subscription to have a specific queue name, and my consumers dont know the names of the queues, nor do they need to know this.

why am I doing this? two reasons:

  1. I can have N of these dynamic categories at a time. I'd like the queue to serve these categories equally. currently we have one queue (msmq) which accepts all of these categories and serves them in FIFO (which means that some categories are starved for some time).

  2. Being able to serve all categories equally, rather than fifo, lets me come up with interesting QoS (by default I understand that Rabbit will round-robin serve messages).

so, back to my question (if its valid): is it possible to consume messages from a queue?

Matteo
  • 37,680
  • 11
  • 100
  • 115
Oren Mazor
  • 4,437
  • 2
  • 29
  • 28

2 Answers2

6

If having dynamic subscribers is an option, I'd like to suggest two possible solutions:

Solution 1:

  • Use a topic exchange.
  • Use the mandatory delivery flag when publishing messages to your exchange.
  • If a message is rejected: create a queue, bind it for the particular routing key and start a subscriber on the queue, the re-publish the rejected message.
  • Use auto-deleting queues so when subscriber stops, their queues will disappear and the whole process of auto-creation can restart.

Solution 2:

  • Use a topic exchange.
  • Use the immediate delivery flag when publishing messages to your exchange.
  • If a message is rejected: create a queue, bind it for the particular routing key and start a subscriber on the queue, the re-publish the rejected message.
  • Use persistent queues. Since queue creation is idempotent, it's OK to go through the re-creation procedure, the essential part here is ensuring a live subscriber is listening on the queue.

If dynamic subscribers are not an option, then here is what I propose. Assuming you're limited to n subscribers:

  • Define a strategy that hashes the dynamic categories in n routing keys,
  • Use a direct exchange,
  • Bind n queues on it for the n routing keys,
  • Have one subscriber per queue.
David Dossot
  • 33,403
  • 4
  • 38
  • 72
  • 1
    these are both interesting options, and I'm reading up more on topic exchanges, but my main issue here is that my subscribers are a) static (i.e. they dont spin up. they're always waiting, like chuck norris), and b) I dont want my subscribers to know of a queue. I want them to equally serve all of my queues – Oren Mazor Dec 02 '11 at 20:39
  • 1
    Sorry I misunderstood then. Messages are accumulated in queues so you have to have queues. Now if the design you're envisioning is a bounded set of consumers pulling from an unbounded number of queues, then read the discussions here: http://stackoverflow.com/questions/8301841/worker-pools-and-multi-tenant-queues-with-rabbitmq/8301978 – David Dossot Dec 02 '11 at 22:19
  • nice answer on that one! but the problem is that even with one queue and n workers, one of his customers can create a thousand tasks and that queue is now blocked until his worker pool can catch up? – Oren Mazor Dec 02 '11 at 22:33
  • I have reviewed my answer to cover the case of non-dynamic subscribers. – David Dossot Dec 02 '11 at 22:33
  • I'm actually trying to figure out if I can have my worker pool listen to one static queue, while the producers continue creating their own anonymous queues. while in the meantime, rabbitmq round-robins those anonymous queues into my static queue. make sense? – Oren Mazor Dec 02 '11 at 22:34
  • Regarding your idea of round robin queues, I'm wondering if the proprietary exchange-to-exchange binding that RabbitMQ support (http://www.rabbitmq.com/extensions.html) could allow you to achieve something similar. Now you'd have to consider the pros/cons of stepping out of pure AMQP... – David Dossot Dec 02 '11 at 22:43
0

With AMQP you publish messages to an Exchange and you consume messages from a Queue. Don't worry about what "queue" means in other messaging technology.

It sounds to me like your scenario could be handled easily with a topic exchange. Publish messages with routing keys like cat.silly, cat.older, cat.interesting. Then have the consumers each declare a queue using the binding key cat.*

This way, all messages published to the exchange with any prefix will be copied to the queue because of the wildcard in the binding key. If your consumers are in fact doing round-robin sharing, i.e. messages should not be copied to multiple queues, then just have all consumers use the same queue name. If every consumer uses the same queue name then you can compile it into your code and not worry about what the name is. But when you want to debug the message flow, just create a consumer that subscribes to a queue named catdebug with the same binding key, cat.*

But if each consumer is specialised and wants to pick and choose which messages to handle, then have each consumer use a unique queue name. That way each consumer will see a copy of every message.

Topic exchanges are the best solution to try first, because the semantics of direct and fanout exchanges can easily be emulated.

Michael Dillon
  • 31,973
  • 6
  • 70
  • 106