Can one SQS poller, handle multiple SQS queues and prioritize it?

Question

I have a use-case where I have 2 SQS queues, each are getting messages. But the condition is if I am getting messages in the FIRST queue, I would stop reading messages from SECOND queue and give priority to the FIRST queue message. Can I handle this using just one SQS poller?

Problem Statement: I often get messages in the FIRST queue, but those are important messages. But I constantly keep on getting messages in the SECOND queue. So, if the poller is processing the message of the SECOND queue, and all of a sudden if a new message comes up in the FIRST queue, then after processing the message of the SECOND queue, it should jump to the FIRST queue and start taking the messages from there until the FIRST queue is empty.

Is there a solution to it?

Answer 1

This is a good way to process priority messages.

If you have just one consumer (you call it a 'poller') of messages, then it should:

• Retrieve and process message(s) from the Priority queue
• If that queue is empty, then retrieve and process message(s) from the Normal queue

You would need to code this behaviour into your consumers. There is no in-built capability to poll from multiple queues.

If you are running multiple consumers, then you might want to have every consumer do the above, or you could assign a portion of them to only process from a specific queue. For example:

• Some consumers could only poll from the Priority queue, using Long Polling to reduce the number of calls but respond as soon as a message appears in the Priority queue
• Other consumers could only poll from the Normal queue, to ensure that these messages are receiving attention too
• You could adjust the relative number of consumers based upon typical data volumes

Alternatively, you could decide to use AWS Lambda functions to process your messages. In this case, queues would be handled in parallel, but without giving one queue priority over the other. The only way to assign priority in such a situation would be to assign different concurrency values to the Lambda functions being used for each queue. This would give the Priority queue more concurrency in case of capacity limits being reached.