RabbitMQ keep processing order of jobs in a same group

Question

I have a list of jobs in my application queue(RabbitMQ).

Some of these jobs are group together and must do in the order.(not continuous, but by order of dispatch time)

For example, consider this 4 jobs in the queue:

[
    { "group": "x", "dispatched_timestamp": 10001, "field1": "some data", "field2": "some other data"},
    { "group": "g", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
    { "group": "x", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"},
    { "group": "t", "dispatched_timestamp": 10005,"field1": "some data", "field2": "some other data"}
]

I must sure the first job in group "x" execute successfully before the thirth job(same group). But i don't care if the fourth job execute sooner than the first(or whatever).

Because sometimes it may happen which all three job deliver to 3 consumer, but the first job fail for some reason(but the second and thirth job has done successful).

I know with this conditions there will be some situations which all jobs in the queue are belongs to same group, so multiple consumers can't work on them and they must deliver one by one. that's ok.

@sonus21 how can it work for grouping jobs? – Amin Shojaei Jun 13 '20 at 11:09 — Amin Shojaei, Jun 13 '20 at 11:09

sonus21 · Accepted Answer · 2020-06-16T12:25:42.503

There's no such thing in AMQ protocol that can lead to this exact solution, there're some ways to solve this problem.

Define queue for each message group
Set concurrency as 1

Let me quote the message ordering from the doc

Section 4.7 of the AMQP 0-9-1 core specification explains the conditions under which ordering is guaranteed: messages published in one channel, passing through one exchange and one queue and one outgoing channel will be received in the same order that they were sent. RabbitMQ offers stronger guarantees since release 2.7.0.

Ref: https://www.rabbitmq.com/semantics.html

First of the foremost things for you is to preserve the message ordering, once we have ordered messages we can utilize the concurrency to handle the messages in order.

Let's say your queue has 5 messages as shown

Queue: Queue1
       +--------------+
Head-->|m1|m2|m3|m4|m5| <---- Tail
       +--------------+

There's the concept of competing consumers, competing consumers means there're more than consumers/subscribers for the same queue. If there is more than one consumer than each of them will run autonomously, which means ordering on the consumer side won't be preserved. To preserve the ordering on consumer side, we should not use competing consumers.

Even though now consumers are not competing, we can still lose message ordering, if we have more than one executor. More than one executor simply means we can poll the queue, send a polled message to any of the executors. Based on the CPU execution policy etc we will still lose the ordering, so now we need to restrict the number of executors to 1.

As we have only one executor each of the polled messages will be executed in orders, so it will become a serial execution.

For Queue1

The executor will consume the message in the following order

-> m1
-> m2
-> m3 
-> m4
-> m5

Still, there's one missing piece, what happens if the execution of m1 is failing? You can retry for N number of times before consuming the next message, to achieve this don't acknowledge unless you have successfully executed any polled message.

From design points of view, this does not look good, since you're processing messages in serial instead of parallel, though you don't have any other alternatives.

RabbitMQ keep processing order of jobs in a same group

1 Answers1