0

I would like to know if I can have persistence in my Spring Integration setup when I use a aggregator, which is not backed by a MessageStore, by leveraging the persistence of AMQP (RabbitMQ) queues before and after the aggregator. I imagine that this would use ack's: The aggregator won't ack a message before it's collected all the parts and sent out the resulting message. Additionally I would like to know if this is ever a good idea :)

I am new working with queue's, and am trying to get a good feel for patterns to use.

My business logic for this is as follows:

  • I receive a messages on one queue.
  • Each message must result in two unrelated webservice calls (preferably in parallel).
  • The results of these two calls must be combined with details from the original message.
  • The combination must then be sent out as a new message on a queue.

Messages are important, so they must not be lost.

I was/am hoping to use only one 'persistent' system, namely RabbitMQ, and not having to add a database as well.

I've tried to keep the question specific, but any other suggestions on how to approach this are greatly appreciated :)

bluemind
  • 1,591
  • 11
  • 16

1 Answers1

1

What you would like to do recalls me Scatter-Gather EI Pattern.

So, you get a message from the AMQP send it into the ScatterGather endpoint and wait for the aggregated reply. That's enough for to stick with the default acknowledge.

Right, the scatterChannel can be PublishSubscribeChannel with an executor to call Web Services in parallel. Anyway the gatherer process will wait for replies according the release strategy and will block the original AMQP listener do not ack the message prematurely.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • The prefetch on the inbound channel must be at least as much as the expected messages in the group. The broker won't send more than prefetch unack'd messages. – Gary Russell Nov 21 '17 at 18:55
  • @GaryRussell If I understand your comment correctly, if I would have outbound requests for multiple original messages (trying to handle several in parallel), my prefetch would have to scale accordingly? – bluemind Nov 22 '17 at 11:18
  • Exactly, yes; container `concurrency * prefetch` would determine the number of outstanding messages allowed by the broker - it's a bit brittle - if you get it wrong the app will hang. I am not very keen on the design. – Gary Russell Nov 22 '17 at 13:57
  • @GaryRussell Is it the design of the 'prefetch', or the design described in the question/answer you do not like? :) What would you use for an alternative? :) – bluemind Dec 04 '17 at 08:41
  • The use of the rabbitmq queue as a "store" is a little odd and, as I said, brittle if you get the settings wrong. I suppose you could deal with that using the group timeout feature and 'nack' the discarded records. – Gary Russell Dec 04 '17 at 17:05
  • The alternative could be a persistent store (JDBC, Redis...). But if the timeout is short, I'm not convinced it is better to use a persistent store whereas Rabbit will keep the messages until they are acked – Nicolas Labrot Jan 02 '18 at 14:19