I have a situation like this:
One single Producer executes many DB queries - the results are sent as messages and arrive to the queue of the single consumer (messages arrive with no special order )
On the other side - a single consumer reads these messages and wants to start processing the messages only when all messages arrived.
What is the best way to solve this kind of a problem?
Is there a particular reason to delay processing until all have arrived?
One method that I've used quite successfully in the past is to dequeue the message and persist the relevant data and sequence number from the message into local storage. Furthermore, as each message arrives, I increment the number of messages received and the total number of messages expected in storage. Finally, all messages have arrived and the increment counter equals the expected number of messages, I perform the actual work.
In this way, we are able to receive all of the messages, but only perform the real work as per your requirements.
