1

I have one sqs-fifo receiving payment notifications from another service that insert items into this sqs-fifo queue.

I also have one lambda that is configured to be triggered by the sqs-fifo (lambda is not polling via code but configured in aws itself to be triggered by the sqs)

The messages are inserted into the sqs-fifo with MessageGroupId=orderId

I still see that sometimes different lambdas handle different notifications in parallel for exactly same orderId.

Example of what I have now:

Time  Payload
00:00 {orderId:1,status:pending} -> lambda request id 222aaa
00:01 {orderId:1,status:charged} -> lambda request id someuuid

Example of what I want to have, version 1 a)

Time  Payload 
00:00 {orderId:1,status:pending} -> lambda request id 222aaa
above call finishes at 00:02, than starts the second request
00:02 {orderId:1,status:charged} -> lambda request id someuuid

Example I want to have, version 2 - batch records for same orderId in same lambda a)

Time  Payload
00:00 [{orderId:1,status:pending}, {orderId:1,status:charged}] -> lambda request id whatever

Is there any way to enforce that these messages are processed one by one via the aws lambda ?

It is very important to avoid race conditions regarding the order of processing payment events for the same orderId

To summarise what I want to achieve:

I want to handle notifications coming for same orderId X only by one particular lambda and avoid more than 1 lambda in parallel being executed for notifications regarding the order with id X. Meanwhile it is fine to spawn one lambda for notification for order X and another one for order Y. That will allow for scalability as I have lots of orders every minute.

Kristi Jorgji
  • 1,154
  • 1
  • 14
  • 39

1 Answers1

1

It does not restrict the messages to "one particular lambda". It does however restrict the messages such that only one running Lambda invocation at a time would ever be processing a message with that group ID.

In other words, it does what you want (prevent more than one message with that ID from being processed concurrently), but not in the way you describe it in your question.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • Thank you. That would be fine if it ensures that the messages are processed one after another in whatever lambda. I saw in logs that this was not respected at some cases. I noticed that let say at 00:00 invocation ddd got orderId 1 charged, at 00:01 invocation aaa got orderId 1 pending notification. And first invocation was still in progress ended at 00:03.. I have put this example of what I currently see in my question – Kristi Jorgji Jun 07 '22 at 14:03
  • @KristiJorgji FYI, the AWS blog post [New for AWS Lambda – SQS FIFO as an event source](https://aws.amazon.com/blogs/compute/new-for-aws-lambda-sqs-fifo-as-an-event-source/) describes the expected batching/invocation behaviour, consistent with Mark B's answer. – fedonev Jun 07 '22 at 16:16
  • @fedonev thank you I will read that as well. My issue is that in production I see different behavior , not always but mostly during high load – Kristi Jorgji Jun 07 '22 at 16:33
  • Maybe show your code where you are setting the MessageGroupID? Right now your question just shows the message payload. You might also log the MessageGroupID attribute on the message that you receive in your Lambda to verify it is there. – Mark B Jun 07 '22 at 16:36