Executing messages in same sequence after getting the messages from sqs fifo queue

Question

I am defining a message listener similar to what is defined below. The messages are query statements. The messages are pushed in a sequence into the queue so that it doesn't violate any foreign key rules when pushed to the tables. When I set the maxnumberofmessages to 10 , I see that the queries seem to be executed out of order. However when I set it to 1 , I don't see any issues. How can ensure the messages are read in the same sequence as they are in the queue when I set maxnumberofmessages to a higher value than 1?

@Bean
public SimpleMessageListenerContainer simpleMessageListenerContainer(AmazonSQSAsync amazonSQSAsync) {
    SimpleMessageListenerContainer simpleMessageListenerContainer = new SimpleMessageListenerContainer();
    simpleMessageListenerContainer.setAmazonSqs);
    simpleMessageListenerContainer.setMessageHandler(queueMessageHandler());
    simpleMessageListenerContainer.setMaxNumberOfMessages(10);
    return simpleMessageListenerContainer;
}


@SqsListener(value = "${sqs.url}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void onMessage(String serviceData, @Header("MessageId") String messageId, @Header("ApproximateFirstReceiveTimestamp") String approximateFirstReceiveTimestamp) {
    repository.execute(serviceData);

}

Modified Code

@Bean
public SimpleMessageListenerContainerFactory simpleMessageListenerContainerFactory() {
    SimpleMessageListenerContainerFactory msgListenerContainerFactory = new SimpleMessageListenerContainerFactory();
    msgListenerContainerFactory.setAmazonSqs(amazonSQSAsyncClient());
    msgListenerContainerFactory.setAutoStartup(false);
    msgListenerContainerFactory.setMaxNumberOfMessages(10);
    msgListenerContainerFactory.setTaskExecutor(threadPoolTaskExecutor());
    return msgListenerContainerFactory;
}


@Bean
public ThreadPoolTaskExecutor threadPoolTaskExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(1);
    executor.setMaxPoolSize(1);
    executor.setThreadNamePrefix("queueExecutor");
    executor.initialize();
    return executor;
}

@SqsListener(value = "${sqs.url}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
public void onMessage(String serviceData, @Header("MessageId") String messageId, @Header("ApproximateFirstReceiveTimestamp") String approximateFirstReceiveTimestamp) {
    repository.execute(serviceData);

}

score 2 · Accepted Answer · answered Apr 25 '18 at 17:31

2

That's because the logic there is like:

   ReceiveMessageResult receiveMessageResult = getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest());
   CountDownLatch messageBatchLatch = new CountDownLatch(receiveMessageResult.getMessages().size());
   for (Message message : receiveMessageResult.getMessages()) {
        if (isQueueRunning()) {
             MessageExecutor messageExecutor = new MessageExecutor(this.logicalQueueName, message, this.queueAttributes);
             getTaskExecutor().execute(new SignalExecutingRunnable(messageBatchLatch, messageExecutor));
        } else {
            messageBatchLatch.countDown();
        }
  }

Pay attention to the getTaskExecutor().execute(). That's how your messages are shifted to their own thread for execution.

You can consider to reconfigure it into the ThreadPoolTaskExecutor with a pool size as 1. And all your messages will be handled on the same thread, therefore the order will be supplied.

answered Apr 25 '18 at 17:31

Artem Bilan

113,505
11
91
118

Thanks @Artem Bilan. What is happening with the code that I posted? Is it creating multiple threads? – Punter Vicky Apr 25 '18 at 17:33
Correct, the default `ThreadPoolTaskExecutor` is really based on the `maxNumberOfMessages`. So, you have there 10 threads in parallel. – Artem Bilan Apr 25 '18 at 17:35
Awesome. Thank You! – Punter Vicky Apr 25 '18 at 17:36
I had one more question. What does isQueueRunning() signify? – Punter Vicky Apr 26 '18 at 15:38
I assume these are spring libraries, Is it possible to achieve the same using Spring? – Punter Vicky Apr 26 '18 at 15:51
The `SimpleMessageListenerContainer` has a functionality like `stop(String logicalQueueName)`. So, you can stop consuming from some specific SQS queue. Sorry, the second your question isn't clear. Right, everything we are talking about here is Spring. – Artem Bilan Apr 26 '18 at 16:05
Sorry , I thought this implementation is used to fetch messages by directly using aws java sdk. How should I integrate the code that you had posted with SimpleMessageListenerContainer? – Punter Vicky Apr 26 '18 at 16:09
??? The `SimpleMessageListenerContainer` has `setTaskExecutor(AsyncTaskExecutor taskExecutor)`. Am I missing anything from your original request ? – Artem Bilan Apr 26 '18 at 16:11
Thanks , I got it! – Punter Vicky Apr 26 '18 at 16:12
In my case , I am receiving messages in my onMessage method which uses SqsListener annotation. If I use this approach , will getAmazonSqs().receiveMessage(this.queueAttributes.getReceiveMessageRequest()); read the messages off the queue? – Punter Vicky Apr 27 '18 at 15:49
??? I don't understand your concern. You still continue to use `@SqsListener`. Only what you need is to inject a proper `ThreadPoolTaskExecutor` into the required `SimpleMessageListenerContainerFactory`: http://cloud.spring.io/spring-cloud-static/spring-cloud-aws/2.0.0.RC1/single/spring-cloud-aws.html#_the_simplemessagelistenercontainerfactory – Artem Bilan Apr 27 '18 at 15:52
Thanks again. I have pasted the modified code to inject ThreadPoolTaskExecutor. Can you please let me know if you see any issues with it? – Punter Vicky Apr 27 '18 at 16:03
`executor.setMaxPoolSize(10);` will lead you out of ordered consumption. I thought you would like to avoid that. Therefore don't configure `ThreadPoolTaskExecutor` for more, than one thread – Artem Bilan Apr 27 '18 at 16:05
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/169960/discussion-between-punter-vicky-and-artem-bilan). – Punter Vicky Apr 27 '18 at 16:06

Executing messages in same sequence after getting the messages from sqs fifo queue

1 Answers1