Spring Integration: TaskExecutor and MaxConcurrentConsumers on AmqpInboundChannelAdapter

Question

My Spring Integration app consumes messages from RabbitMQ transforms them to SOAP message and does web service request.

It is possible to get many (10 – 50) messages per second from the queue. Or after application restart there could be many thousand messages in RabbitMQ queue.

What is the best possible scenario to process up to 10 messages in parallel threads (message ordering is nice to have but not required feature, if web service answers with business failure then failed message should go to retry until it succeeds).

Amqp listener should not consume more messages from the queue as not busy threads available in task executor. I could define an ThreadExecutor in an channel like this:

@Bean
public AmqpInboundChannelAdapterSMLCSpec 
amqpInboundChannelAdapter(ConnectionFactory connectionFactory, Queue queue) 
{
  return Amqp.inboundAdapter(connectionFactory, queue);
}

IntegrationFlow integrationFlow = IntegrationFlows
  .from(amqpInboundChannelAdapter)
  .channel(c -> c.executor(exportFlowsExecutor))
  .transform(businessObjectToSoapRequestTransformer)
  .handle(webServiceOutboundGatewayFactory.getObject())
  .get();

Or is it enough to define an task executor in AmqpInboundChannelAdapter like this and do not define channels task executor in the flow definition:

@Bean
public AmqpInboundChannelAdapterSMLCSpec 
amqpInboundChannelAdapter(ConnectionFactory connectionFactory, Queue queue) 
{
  return Amqp.inboundAdapter(connectionFactory, queue)
             .configureContainer(c->c.taskExecutor(taskExecutor));
}

Or maybe to define task executor for a channel like option 1 but additionally set maxConcurrentConsumers on a channel adapter like this:

@Bean
public AmqpInboundChannelAdapterSMLCSpec 
amqpInboundChannelAdapter(ConnectionFactory connectionFactory, Queue queue) 
{
  return Amqp.inboundAdapter(connectionFactory, queue)
             .configureContainer(c->c.maxConcurrentConsumers(10));
}

score 2 · Accepted Answer · answered Aug 28 '18 at 20:46

2

The best practice to configure a concurrency on the ListenerContainer and let all downstream processes happen on those threads from the container. This way you'll get a natural back-pressure when no more messages are polled from the queue because the thread is busy. And on the other hand there is not going to be message loss just because with an ExecutorChannel after the listener container you're going to free a polling thread and the current message will be acked as consumed, but you may fail downstream.

answered Aug 28 '18 at 20:46

Artem Bilan

113,505
11
91
118

Thank you Artem for your reply. Do you know some URL with an example code which configures a rabbitmq client app consuming more than one queue with concurrency and retry functionality? – Roman Aug 29 '18 at 06:32
The `Amqp.inboundAdapter()` can accept several queues. Concurrency is the matter of the `configureContainer()`. And retry can be applied via `adviceChain(Advice... adviceChain)` in the mentioned `configureContainer()`. How to configure and appropriate retry advice, you can read in the docs: https://docs.spring.io/spring-amqp/docs/2.0.5.RELEASE/reference/html/_reference.html#retry – Artem Bilan Aug 29 '18 at 13:59

Spring Integration: TaskExecutor and MaxConcurrentConsumers on AmqpInboundChannelAdapter

1 Answers1