Understanding SqsMessageDrivenChannelAdapter behaviour

Question

I'm trying to understand the behavior of SqsMessageDrivenChannelAdapter to address memory issue. The upstream system dumps thousands of messages in aws-sqs-queue, all of the messages are received immediately by SqsMessageDrivenChannelAdapter. On the AWS console I do not see any messages available on the queue. The SqsMessageProcesser then processes 1 message every 5 seconds.

Here's the log:

2019-05-21 17:28:18 INFO SQSMessageProcessor:88 - --- inside sqsMessageProcesser--- 2019-05-21 17:28:23 INFO SQSMessageProcessor:88 - --- inside sqsMessageProcesser--- 2019-05-21 17:28:28 INFO SQSMessageProcessor:88 - --- inside sqsMessageProcesser--- 2019-05-21 17:28:33 INFO SQSMessageProcessor:88 - --- inside sqsMessageProcesser--- 2019-05-21 17:28:38 INFO SQSMessageProcessor:88 - --- inside sqsMessageProcesser--- .........................

Does this mean that while SqsMessageProcesser is processing 1 message every 5 seconds, thousands of messages are being held in (server) memory of the in-channel?

Each db transaction takes around 5 seconds and currently we are facing 'outofmemory' issues on PRD.

Will it help if i set the capacity on the QueueChannel and setMaxNumberOfMessages for SqsMessageDrivenChannelAdapter? If yes, is there a standard way to calculate these values?


  @Bean(name = "in-channel")
  public PollableChannel sqsInputChannel() {
    return new QueueChannel();
  }

  @Autowired
  private AmazonSQSAsync amazonSqs;

  @Bean
  public MessageProducer sqsMessageDrivenChannelAdapterForItems() {

    SqsMessageDrivenChannelAdapter adapter =
        new SqsMessageDrivenChannelAdapter(amazonSqs, "aws-sqs-queue");
    adapter.setOutputChannelName("in-channel");
    return adapter;
  }

  @ServiceActivator(inputChannel = "in-channel",
      poller = @Poller(fixedRate = "5000" , maxMessagesPerPoll = "1"))
  public void sqsMessageProcesser(Message<?> receive) throws ProcesserException {
  logger.info("--- inside sqsMessageProcesser---")
  // db transactions.
}

score 0 · Answer 1 · answered Apr 08 '20 at 20:18

Actually it is an anti-pattern to place a QueueChannel for message-driven channel adapter. The later is already async and based on some task scheduling. So, shifting consumed messages from source into an in-memory queue is definitely leading into some troubles.

You should consider to have a direct channel instead and let SQS consuming thread to be blocked until your sqsMessageProcesser finishes its job. This way you will guarantee no data loss.

Understanding SqsMessageDrivenChannelAdapter behaviour

1 Answers1