4

I have been searching for the correct messaging pattern to use for the following problem:

I have one queue that has messages for every userid in the domain, each message is a userChanged event. The business requirement is that all messages for a particular userid must be processed in FIFO order and if an error occurs during the processing of a particular userid's message, no more processing should occur until that error is marked as processed successfully. All messages must be processed and this solution needs to be deployed in a clustered ESB environment.

I would like to demux these events into a FIFO queue for each userid so I can process messages from different userids in parallel and process each message for a user sequentially.

So far, I have come up with two possible solutions involving Mule and Rabbit but both would involve either custom components or existing components that I am not aware of.

  1. Have all the messages on the first RabbitMQ queue read by a Mule flow with an AMQP inbound endpoint that grabs the userid from the header. It then uses an AMQP outbound endpoint which dynamically creates a persistent queue if it doesn't exist already that is named something like: userChangedEvent-#[flowVars.userid] and publishes the message to that queue. The custom component(s) would need to do the following: a. Create a shared object map that would identify all the dynamic amqp listener flows there are if it doesn't exist b. Check the map to see if there is an instance of the listener flow for the userid. If there is not, add an instance of the dynamic amqp listener flow that will listen to the userChangedEvent-#[flowVars.userid] queue, add this to the map using the userid as the key. c. Start the flow. d. The dynamic flow would need to be configured to process in a single threaded configuration, manually ack the messages after the business logic was completed successfully and to stop the flow in the case of an error.

  2. Have one amqp inbound endpoint reading all the messages off the queue and then a "collating" router would route messages with the same userid to the same instance of the reusable business logic flow for that userid, creating an instance if one for a given userid doesn't exist. Outstanding questions for this scenario are: a. Does such a router exist or would it need to be developed? b. The business logic flow would need to be executed in a single thread so the backlog of messages would need to be maintained in a durable fashion for each instance of the flow. c. An error in execution of one of the instances should stop that instance from processing more messages until it is resolved.

I have thought of workarounds such as "bucketing" the userids into various predefined queues (userChangedEvent-0to1000, userChangedEvent-1000to2000, etc) so that we can predefine all the flows and associated amqp listeners we need and eliminate the need for dynamic flows, but this strikes me as an inelegant solution.

I get the feeling that there must be a messaging pattern to address this but I ripped up my copy of EIP to no avail! Any suggestions would be appreciated.

Update: This is conceptually what I am after, a demuxer with lanes (what I was calling buckets) but I was thinking that dynamic lane creation (1 per userid) would be even better: http://www.coralblocks.com/index.php/2014/06/demultiplexing-with-coralqueue-for-parallel-processing/

DavidFry
  • 43
  • 5

1 Answers1

1

There is no way you can have multiple concurrent consumers and honor fifo ordering at the same time. You can, however, have multiple non-concurrent consumers for an improved availability using a feature called: exclusive consumers. You can activate this with the attribute exclusiveConsumers of the connector.

Regarding the stop processing on error, I can suggest two different approaches:

  • If this business requirement is likely to change, you could leverage a complex event processing engine as a semaphore for events.
  • Otherwise, you could use the circuit breaker pattern as described here, this is probably the easiest way.
Víctor Romero
  • 5,107
  • 2
  • 22
  • 32
  • Thanks Victor for the suggestions, circuit breaker looks promising and the CEP approach sounds interesting - do you have any docs to point to for that? – DavidFry Dec 04 '14 at 17:05
  • I probably wasn't clear with my initial question. I am not expecting to get multiple concurrent consumers on the same queue and respect fifo ordering. I want to take the stream of messages that are coming off the first queue and separate them by userid into flows that will process in fifo order and make use of the circuit breaker pattern to stop processing for a given userid if there is an exception. I was suggesting demuxing with queues (one queue per userid) or by using a stateful router. – DavidFry Dec 04 '14 at 17:10
  • Rabbitmq will honor insertion order, so default configuration should be fine. Documentation around the circuit breaker can be found in the here workd int he aswer. Examples on esper are http://blogs.mulesoft.org/mule-esper-cep-and-non-events/ and http://www.manning.com/dossot2/MuleinAction2E_CH14.pdf – Víctor Romero Dec 04 '14 at 17:52
  • Thanks for the links Victor. Please look at the update where I posted a line showing a Demuxer with lanes, this is really functionally what I am after: This is conceptually what I am after, a demuxer with lanes (what I was calling buckets) but I was thinking that dynamic lane creation (1 per userid) would be even better: http://www.coralblocks.com/index.php/2014/06/demultiplexing-with-coralqueue-for-parallel-processing/ – DavidFry Dec 04 '14 at 18:43
  • Personally I would use something like ActiveMQ's message groups ( http://activemq.apache.org/message-groups.html ) however that is now available on rabbitmq ( http://stackoverflow.com/questions/20530591/message-groups-in-rabbitmq-amqp ). With what you have, I would probably receive the messages in one queue, put the elements in a different one with just one header enritchment with the bucketname you want. And then I would leverage a message selector on a differnt exclusive inbound endpoint. – Víctor Romero Dec 04 '14 at 19:10
  • Thanks Victor, Message Groups is exactly what I was looking for. Just as an aside, it seems that not using a message selector and relying on the broker to assign groups to consumers automatically would allow for dynamic bucket creation down to the custid level if needed: http://activemq.apache.org/how-do-message-groups-compare-to-selectors.html – DavidFry Dec 04 '14 at 20:44
  • Yes absolutely, sorry I misunderstood and thought that you were tied to rabbit. If you can go with active, then message groups is the way to go! – Víctor Romero Dec 04 '14 at 20:58