C++ client : Ensuring Message Consumption Consistency in Aeron with Multiple Consumers on same channel

Question

I have a scenario where I'm using Aeron messaging library with multiple consumers consuming from the same channel. I want to ensure that each message is consumed by only one consumer to avoid duplication and guarantee consistency across consumers. In Kafka, we have the concept of committing offsets to track the progress of each consumer. Is there a similar mechanism or approach in Aeron to achieve message consumption consistency?

In my setup, I have multiple consumers that subscribe to the same Aeron channel and I want to make sure that messages are distributed across consumers in a balanced manner. I'm aware that Aeron provides a broadcast approach by default, where each consumer receives a copy of every message. However, I'm interested in a scenario where each message is consumed by only one consumer.

I have explored the Aeron documentation and API, but I couldn't find a built-in mechanism similar to Kafka's offset committing. I would like to know if there are any recommended patterns or techniques to achieve message consumption consistency in Aeron with multiple consumers.

My specific requirements are as follows:

Guarantee that each message is consumed by only one consumer to avoid duplication. Ensure consistency across consumers, so that no two consumers process the same message in same channel. Ideally, have a mechanism to track the progress of each consumer, similar to Kafka's offset committing. I would appreciate any insights, recommendations, or best practices for handling this scenario in Aeron. If there are any C++ code examples or configuration suggestions, they would be highly valuable.

Thank you in advance for your help!

Answer 1

TL;DR: This model is not supported by Aeron and you are unlikely to find examples to do something similar.

Forcing Aeron to work this way would prove difficult. E.g. how would you co-ordinate multiple subscriptions to the same multicast UDP channel across separate machines? How do you ensure that each message is actually processed, i.e. what if one of the consumers crashes half way through processing the message? Requiring an infrastructure to provide these strict guarantees is very difficult so many systems tend toward allowing redundant, idempotent operations and at least once delivery. The redundancy of operations provides better reliability and the simpler implementation model will tend to perform better*.

*Stricter guarantees like avoiding duplication will need more communication between to components to ensure that the consistency requirements are met. This leads to higher latency and lower throughput.