Imposing barrier after group of messages

Question

I create groups of messages (e.g. G1, G2,...).

Within each group, all messages can be processed in parallel, order does not matter.

However, to process G2, all messages from G1 must have finished processing.

Is it possible to realize this in rabbitmq? (finally I will implement it in symfony...)

My question is related to Message Queue with 'Message Barrier' Feature?

score 0 · Answer 1 · answered Sep 24 '19 at 02:52

0

Queues are not a good fit for such scenarios. Use an orchestrator like Cadence Workflow which supports your scenario out of the box. It also allows to implement sophisticated error handling and retry policies.

answered Sep 24 '19 at 02:52

Maxim Fateev

6,458
3
20
35

Thank you! I am still hoping to find a "closer" concept to make my existing message queues do what I need. I looked into Cadence, but it is quite a large bundle of appliances that I cannot install "just" to make one cron job more efficient. – olidem Sep 24 '19 at 08:12

score 0 · Answer 2 · answered Oct 23 '20 at 03:03

RabbitMQ supports priority queues, when using them messages with higher priority should be sent to consumers before ones with lower priorities (some caveats apply).

When dispatching a message, you can set its priority by using a AmqpStamp.

$bus->dispatch($message, [new AmqpStamp($routingKey, $flags, ['priority' => 10])]);

Imposing barrier after group of messages

2 Answers2