0

Let's say I have a FIFO SQS, and a lambda which consumes a batch of messages from the FIFO SQS. The max limit on the size of this batch is 10, as mentioned here. I was wondering how this would work.

Let's say we have some message group ID G, because of which we have 5 messages groups - G1, G2, ..., G5.

Let's say we have set the batch size as 10. So that would mean at a time, more than one message would be required to be picked up from each group. So let's say we have two messages M1 and M2 inside G1. M1 came first, and M2 came afterwards.

But since we are getting both of these messages in the same batch, they might get processed out of order.

Is this the correct expectation? Or would the FIFO queue only put 5 messages in the batch, and send that to the lambda?

Mooncrater
  • 4,146
  • 4
  • 33
  • 62

1 Answers1

1

The batch will be supplied to the AWS Lambda function in order.

Let's say that A/B/C represents the Group and the number represents the message number.

Let's assume that the messages were sent in this order: A1 B1 A2 C1 A3 D1 C2 B2 A4 D2 C3 B3

If the batch size is set to 10, then the Lambda function would receive the first 10 messages:

A1 B1 A2 C1 A3 D1 C2 B2 A4 D2

They would be provided in the batch in that order so your code should process the messages in the order given.

While the batch contains multiple entries for Group A, they remain in the order that the messages were sent.

Even though there are extra messages waiting, no other Lambda functions will be triggered because there are unprocessed messages in each Group. However, if a message E1 was sent, then a Lambda function would be triggered with E1 since it can be processed at any time without waiting for earlier messages to be processed.

Once the initial Lambda function has finished running, the function will again be invoked with C3 B3 since the earlier messages in those groups have now been processed.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Hey John! - Thanks for the answer - Any doc where more on this is present? – Mooncrater Aug 04 '23 at 07:29
  • 1
    It mostly comes from playing around with `MessageGroupId`. The main thing to understand is that a new batch of messages will not include a Message Group ID that has an earlier message remaining to be processed. So, let's say there is always a batch size of 1. Using the above examples, Lambda functions would be invoked with `A1 B1 C1 D1`. Only when A1 has finished processing will A2 be provided. However, with a larger batch size, it _will_ provide multiple messages in the same group. – John Rotenstein Aug 04 '23 at 08:09
  • Got it - Thanks! – Mooncrater Aug 04 '23 at 08:23