0

We have provisioned a single AWS SQS FIFO queue. There is a single process that adds items to this queue. All items added have the same group message id.

We start two independent identical processes, Consumer A and Consumer B. The only thing the consumers do is pull items off the queue and throw away the results. Assume that there are no network or service interruptions on either the AWS end or at our end.

I have looked carefully through AWS's documentation and cannot find an answer to this question: When can two AWS SQS FIFO queue consumers process messages with the same group message id?

I did see this text:

When messages that belong to a particular message group ID are invisible, no other consumer can process messages with the same message group ID.

Does the above apply only when all messages with the same message group ID are invisible? Or is it enough for just some of the messages for a given group message id be invisible.

For example, imagine that when the above two consumers start there are already 10,000 messages on the queue all with the same group message id. Since a consumer can be sent a maximum of 10 messages at once, does this mean that while consumer A is processing 10 messages consumer B can't get any messages?

I am looking for a reference in the AWS documentation that clarifies this, or perhaps, someone has done an experiment that decides this one way or the other.

rlandster
  • 7,294
  • 14
  • 58
  • 96
  • "When messages that belong to a particular message group ID are invisible" means 1 or more messages. Consumers A and B cannot concurrently processes messages from this queue because, as you've indicated, all message have the same message group ID. To leverage any concurrency, your messages would need 2+ unique message group IDs. – jarmod Dec 05 '22 at 18:20
  • See [here](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html): "When you receive a message [or batch of messages] with a message group ID, no more messages for the same message group ID are returned unless you delete the message or it becomes visible." – jarmod Dec 05 '22 at 18:24

1 Answers1

3

If any single message of a group is invisible (in-flight / currently being processed) no other consumer can process / receive any message(s).

That is the only way to ensure in-order / FIFO processing because if the processing of the currently invisible message fails it needs to be reprocessed and it must not happen that another message within the same group was already processed in the meantime.

Don't have a doc for that because in my mind this is obvious and the only logical thing to do. Anything else will break the FIFO principle.

And to answer "When can two AWS SQS FIFO queue consumers process messages with the same group message id?" - never. Unless the visibility timeout of one message expires, it becomes visible again and the second consumer picks it up. But at that point from the perspective of the queue only the second consumer is working on the message, the first does not count anymore.

luk2302
  • 55,258
  • 23
  • 97
  • 137