0

I have a use case where I have to use a queuing mechanism that makes sure messages are processed by a consumer ("workers") in order and one after another.

I have used RabbitMQ in the past and it guarantees the order in which it has received the message. But what if that order is not correct?

Let's say I submit messages [4, 5, 3, 2, 1], RabbitMQ consumers will process the messages in that order. What if I want them to be processed in the order [1, 2, 3, 4, 5] because the messages depend on each other?

Also, I don't want to allow a consumer to consume a message 3 before message 2 has been acknowledged (no gaps).

Are there any queueing solutions that support this use case? Currently, we dump the messages into a database and have a worker pull the data periodically in order.

mitchkman
  • 6,201
  • 8
  • 39
  • 67

1 Answers1

1

Consider one of the 2 patterns: "Message Sequence" or "Resequencer".

A Message Sequence is useful as follows:

Whenever a large set of data may need to be broken into message-size chunks, send the data as a Message Sequence and mark each message with sequence identification fields.

A Resequencer is a bit different:

Use a stateful filter, a Resequencer, to collect and re-order messages so that they can be published to the output channel in a specified order.

The Resequencer can receive a stream of messages that may not arrive in order. The Resequencer contains in internal buffer to store out-of-sequence messages until a complete sequence is obtained. The in-sequence messages are then published to the output channel. It is important that the output channel is order-preserving so messages are guaranteed to arrive in order at the next component. Like most other routers, a Resequencer usually does not modify the message contents.

I would highly suggest reading through these patterns in the book "Enterprise Integration Patterns" by Gregor Hohpe/Bobby Woolf (with contributions from the likes of Martin Fowler et al.)

There's more detail than I can describe here, but essentially a message sequence depends on having a sequence identifier, position, and "end" (boolean field). You'll need an adapter on the end of the queue to process the sequence.

A sequencer, on the other hand, "stores out-of-sequence messages in an internal buffer until a complete sequence is obtained, and then publishes the messages to the output channel in the proper sequence" (p. 285).

This is pretty similar to your "dump to db, then have a worker pull" strategy that you're currently using.

You implementation details of these patterns will be based on your application language and choice of queue (RabbitMQ in your case), but the patterns are already pretty well-established, so I would take a good look at those.

I'm unaware of any built-in mechanism in RabbitMQ itself that can help you obtain this.

Hope this helps.

Edit

I googled "RabbitMQ resequencer" and found the following (can't vouch for the efficacy though): rabbus-sequence (GitHub).

In any case, might help to look at whatever their code is doing there to see if it's something you can get some inspiration from.

Community
  • 1
  • 1
Josh Beam
  • 19,292
  • 3
  • 45
  • 68