Deleting messages from SQS queue after processing

Question

My application sends messages to a AWS SQS queue for jobs that need some sort of background processing. My processing daemon receives and processes the messages like this:

$result = $sqsClient->receiveMessage([
    'QueueUrl' => \Myapp\Config::get('sqs.queue'),
    'WaitTimeSeconds' => 20,
    'MaxNumberOfMessages' => 1
]);

if (isset($result['Messages'])) {
    foreach ($result->getPath('Messages/*/Body') as $messageBody) {
        $handler = new \Myapp\Handler();
        $handler->dispatch($messageBody);
    }
}

This works fine, however I can see from the SQS console that the messages that my script retrieves are put into the "Messages in Flight" category, and then after a while they're put back into "Messages Available", and my script ends up picking them up again.

How can I remove the messages from the SQS queue, or even better mark them as completed?

score 44 · Accepted Answer · answered Feb 22 '15 at 01:00

When you receive a message from an SQS queue, the message will (by default) return to the queue 30 seconds later. This is to handle cases where processing of the message crashes and the message needs to be processed again.

Once your message is successfully processed, use deleteMessage to delete the message. You'll need the receiptHandle value from the message when you received it from receiveMessage in order to delete the message.

If typical processing of your message may take more than 30 seconds, then you can configure your queue to increase that "return to queue" time. This is called "Default Visibility Timeout" in the SQS queue configuration.

Also be aware that Amazon SQS works in such a way that:

messages may be received out-of-order compared to how they were added to the queue
messages may be received twice, so allow your message processor to handle these cases

Didn't know about the possibility of messages being received twice, thanks for that. Great answer. — John Dorean, Feb 24 '15 at 23:17
Worth noting, there are no duplicate messages in FIFO queues - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html — slajma, Feb 14 '21 at 03:06

score 4 · Answer 2 · answered Feb 22 '15 at 11:12

4

You do need to delete messages after you process them - that would be considered 'marking them completed'; however, if something downstream needs to act on this 'completed'' action, its not uncommon for me to post a 'completion' message to another SQS queue, and then delete the message from the incoming queue. In this way another process, either now or down the road can take action that needs to be done when the first worker completes its work, i.e. chaining those processes together in a decoupled manner.

If there is nothing downstream that needs to be done, simply deleting it is enough.

answered Feb 22 '15 at 11:12

E.J. Brennan

45,870
7
88
116

1

In most cases you really do need to delete the message after you process it. e.g. 1) if you are simply reading from the queue via your code and you don't want the message to pop back up in the queue and be re-read later, which it will do if you don't delete it, and 2) if you are allowing batches of sqs messages to trigger lambas, then you need to delete messages that were processed successfully to avoid every message in the batch from retriggering the lambda (you normally only want the failures to retrigger the lambda) – adavea Feb 28 '21 at 20:36

Deleting messages from SQS queue after processing

2 Answers2