1

I've a FIFO SQS queue with a lambda polling it. The lambda had this error for one of the msgs of queue:

Runtime exited with error: signal: killed Runtime.ExitError

I know this happened due to max memory reached.

Now, any msg sent to queue after this error stays in queue with the faulty msg stuck in Flight as seen on the AWS console.

What I expected: Msg with max memory reached should have exited, allowing next msgs to be in flight.

What happened: AWS console showed 1 in flight msg and the new msgs sent after delay were shown under "Messages available" until I had to purge the queue to empty it.

I'm trying to understand if runtime is killed, does the msg is kept in flight permanently?

Lambda is at default concurrency capacity.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Waleed Ahmad
  • 446
  • 3
  • 15

1 Answers1

2

Since you are using an Amazon SQS FIFO queue, it sounds like all of your messages have the same MessageGroupId.

When using a FIFO queue, SQS will ensure that all messages with the same MessageGroupId will be processed in order. If the first message is not being correctly processed, it will not send any other messages with the same MessageGroupId for processing, since they will be out-of-order.

If you do not strictly need these messages to be processed in order, then you could supply different values for MessageGroupId, or even use a standard (non-FIFO) queue.

To fix your current situation, you could activate a Dead Letter Queue on this SQS queue, configured to send any messages with repeated failures to an alternate ('Dead Letter') queue. This would then allow subsequent messages to be processed.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Thanks John, That makes perfect sense with messing up the order. However, I'm still unclear why faulty message is stuck under the "Messages in Flight". Since the logs show lambda exited with error message, why that message is still in flight? – Waleed Ahmad Nov 04 '22 at 06:13
  • It is possibly being continually reprocessed by Lambda since a Dead Letter Queue has not been configured. Please note that the SQS console just calls `ReceiveMessage()` just like any consumer of SQS messages, so viewing a message in the console causes the message to be 'in flight' until the message is deleted or its invisibility duration expires. – John Rotenstein Nov 04 '22 at 06:53