2

Quite new to RabbitMQ and I'm trying to see if I can achieve what I need with it.

I am looking for the Worker Queues pattern but with one caveat. I want to have only a single worker running concurrently per routing key.

An example for clarification:

If i send the following messages with routing keys by order: a, a, b, c, I want to have only 3 workers running concurrently. When the first a message is received a worker picks it up and handles it.

When the next a message is received and the previous a message is still handled (not acknowledged) the new a message should wait in queue. When the b and c messages are received they each get a worker handling them. When the first a message is acknowledged any worker can pick up the next a message.

Would that pattern be possible using RabbitMQ in a natural way (without writing any application code on my side to handle the locking and stuff...)

Edit:

Another clarification. All workers can and should handle all messages, and I don't want to have a queue per Worker as I want to share the load between them, and the Publisher doesn't know which Worker should process the message. But I do want to make sure that no 2 Workers are working on messages sharing the same key at the same time.

For example, if I have a Publisher publishing messages with a userId field, I want to make sure no 2 Workers are handling messages with the same userId at the same time.

Edit 2

Expanding on the userId example. Let's say I have a single Publisher and 3 Workers. The publisher publishes messages like these: { userId: 1, text: 'Hello' }, with varying userIds. My 3 Workers all do the same thing to this messages, so I can have any of them handle the messages coming in. But what I'm trying to achieve is to have only a single worker processing a message from a certain user at the same time. If a Worker has received a message with userId 1 and is still processing it, and another message with userId 1 is received I want to make sure no other Worker picks up that message. But other messages coming in with different userIds should be processed by other available Workers.

userIds are not known beforehand, and the publisher doesn't know how many workers are or anything specific about them, he just wants to schedule the messages for processing.

Netanel G
  • 59
  • 4

1 Answers1

1

what your asking is not possible with routing keys, but is built into queues with a few settings.

if you define "queue_a" for a messages, "queue_b" for b messages, etc, you can then have as many consumers connect to it as you want.

RabbitMQ will only deliver a given message to a single consumer of a given queue.

The way it works with multiple consumers on a single queue is basic round-robin style dispatch of the messages. that is, the first message will be delivered to one of the consumers, and the next message (assuming the first consumer is still busy) will be delivered to the next consumer.

So, that should satisfy the need to deliver the message to any given consumer of the queue.

To ensure your messages have an equal chance of getting to any of the consumer (and are not all delivered to the same consumer all the time), there are a few other settings you should put in place.

First, make sure to set the message consumer no ack setting to false (sometimes called "auto ack"). This will force you to ack the message from your code.

Lastly, set the "consumer prefetch" limit of the consumer to 1.

With this combination of settings, a single consumer will retrieve a single message and begin working on it. While that consumer is working, any message waiting in the queue will be delivered to other consumers if any are available. If there are none available, the message will wait in the queue until a consumer is available.

With this, you should be able to achieve the behavior you are wanting, on a given queue.

...

Keep in mind this only applies to queues, though. routing keys cannot be managed this way. all matched routing keys from an exchange will cause a copy of the message to be sent to the destination queue.

Derick Bailey
  • 72,004
  • 22
  • 206
  • 219
  • In my point of view this awnser is right. But the awnser is even for queues a no, because the question was _"any worker can pick up the next a message"_ and this is not possible. In this scenario every queue has only one worker and one routing key it acts on. So every time the same worker picks up the next a message. – slowjack2k Aug 15 '16 at 06:36
  • Thanks, Derick, but @slowjack2k is correct about what I meant. All my messages for the workers are the same, and I don't care (and don't want to care) about which worker picks up the message. I just want to make sure only a single worker operates on some messages based on some key (that I generate per message). – Netanel G Aug 15 '16 at 10:17
  • @NetanelG these two statements don't go together `All my messages for the workers are the same` and `messages based on some key (that I generate per message)` Aslo Derick's answer is correct for the question you have asked the way you asked it, but based on this comment I think you should edit it. – cantSleepNow Aug 15 '16 at 13:43
  • @cantSleepNow You right about the comment, I didn't express myself correctly there. By all messages are the same I meant they are handled the same way, not that they have the same content. About the question, as slowjack2k said, I stated that any worker can pick up any message, but I'll try to make it clearer in the question. – Netanel G Aug 15 '16 at 15:05
  • ah - you're right. i misread (or forgot about) that part of the question. in that case, you don't need to use the "exclusive" option for a given consumer. i'll update my answer – Derick Bailey Aug 15 '16 at 15:24
  • Nice @DerickBailey It's getting there. Now if we continue on the `userId` example, I don't know all `userId`s at design-time so I can't pre-create a queue for each, but I can create them as new users send messages. Would it be possible for me to create a queue for every `userId`? Can that kind of solution scale, with workers listening on lots of queues at the same time? – Netanel G Aug 16 '16 at 07:30
  • in my experience, queue per user is a bad idea. this tends to group the messages incorrectly - you would send all messages about the user, through that queue, even when the message has to be processed differently than others. this forces you to re-create the logic of RMQ's routing inside your own code, so you can process different types of messages, differently. i would recommend creating queues around message types, not around users. – Derick Bailey Aug 16 '16 at 12:01