5

I am sending messages from an SQS queue to a Node.js Lambda function. I send about 20 messages at the same time to the queue after which Lambda starts working.

Some of these messages never seem to appear in the queue, or at least don't make it to Lambda. I don't see any activity in the Cloudwatch logs for those missing messages.

It's a long polling queue with these settings:

  • Default Visibility Timeout: 2 minutes
  • Delivery delay: 0 seconds
  • Receive Message Wait Time: 5 seconds
  • Queue type: Standard

The Lambda script finished in a few seconds and has concurrency set to 'Use unreserved account concurrency'.

For debugging purposes, I'm now also sending all the messages to a second queue that doesn't get processed. There I can see all the messages, also the ones that are missing from the other one.

The messages just contain about 3 numeric variables in a json, so the size shouldn't be an issue.

I have set up a redrive policy, and the missing messages don't show up in the dead letter queue either.

Kinda pulling my hair out here. Any ideas why these messages seem to disappear?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Rick
  • 628
  • 1
  • 7
  • 20
  • 1
    Could you include the result of logging the event that your lambda is invoked with? – thomasmichaelwallace Nov 16 '18 at 13:38
  • 7
    My guess is multiple messages are being sent to a single Lambda invocation (which is the default setting, and you didn't include the Batch Size setting in your question) and your Lambda function is only handling the first message instead of properly looping through all the messages passed to it. – Mark B Nov 16 '18 at 14:30
  • 3
    Setting the Batch Size to 1 fixes things. – Rick Nov 16 '18 at 20:21
  • 1
    Or you could simply add a loop into your Lambda function, process all N messages sequentially, and hopefully save some time and money. – jarmod Nov 17 '18 at 22:01
  • Good one, I'll do that. – Rick Nov 18 '18 at 19:19
  • you may solve this issue at https://stackoverflow.com/questions/51596709/sqs-triggers-lambda-with-multiple-records-messages – Đào Ngọc Lâm Jul 04 '19 at 04:37

0 Answers0