1

Seeing that some event sources send multiple event records in a batch1, and that a single Lambda may be made available to both to a multiple-record event sources (e.g. SQS, SES) and a single request event source (e.g. API Gateway), does anyone have any examples of handling both a single API request and multiple record SQS or other source events with the same Lambda?

This is a little more involved than limiting the batch size, since the event objects from different services don't always have a list of records, e.g. API Gateway sends a single request object event payload, rather than a list of records.

I have been able to figure out how to parse the events based on the eventSource key; however, I don't understand how Lambda responds to multiple-record events.

Is there one invocation record per event record, or is there a single invocation record, i.e. does the Lambda code run once per batch or multiple times?

Is my Lambda code responsible for forming the invocation record(s), or just the response payload?

How do I form my response payload(s) from a multiple-record event batch?

Do I respond to each record individually and build up a single payload with the responses for each individual record, then return the aggregate payload?

On one hand, I may just want the Lambda to do something and not return anything, such as an asynchronous invocation that isn't expecting a response. On the other hand, a synchronous API gateway request would expect a response, or another asynchronous invocation may expect responses to be published to another queue.

I am making the assumption that each record in a multiple-record SQS Queue event batch is equivalent to a single request event from API Gateway. Is this incorrect?

scottmains
  • 31
  • 5

3 Answers3

0

You will allways get an array of Records, you can see the event structure in the documentation for Lambda

Martin Trenker
  • 86
  • 1
  • 3
  • Thanks. The link answers the main point of the question: "Lambda reads messages in batches and invokes your function once for each batch." I still need some more information about how to send synchronous responses for when more than one record is sent in the batch. I can process the records in a loop, but I can't find an example multi-record response payload, as I have been able to find for [API Gateway response format](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) – scottmains Oct 21 '20 at 22:00
0

You will receive an array of one or more records.

In developing your code always go under the assumption that multiple records are submitted for the request (iterate over them in a loop) and process them one record at a time.

With the batches the services will generally wait a short length of time to see whether the total batch size can be reached, and then forward this as an event to your Lambda.

The API Gateway invocation of Lambda will contain a completely different format for the event as can be seen in this example. This should not be a problem though, as you would generally expect to develop a Lambda for API Gateway, rather than implementing for multiple services at the same time.

Chris Williams
  • 32,215
  • 4
  • 30
  • 68
  • Thanks. I can iterate over the records but if I return after the first record I assume the Lambda will exit after the first record. Do I build a list of responses in the loop then return the list as the payload? – scottmains Oct 21 '20 at 20:57
  • I prefer to use the same Lambda to handle an API Gateway request as SQS messages, since the response logic is the same for the request data regardless of the event source. I can generically parse the event input data from various sources, and I am using the loop approach even for the API events which I put in a list of one request; however, the API requests expect a response payload that is not a list. I want to understand more about how responses for multiple record events batches are typically returned. – scottmains Oct 21 '20 at 21:12
  • It seems like we are saying that the Lambda is invoked once with all records in a batch as a single event payload that has to be handled inside the Lambda code, e.g. in a loop for instance. Lambda does not know how many records are in an event when the function is invoked, and thus cannot magically know how to run multiple times. If this is the case, I suppose a Lambda should typically not return anything until all records in the event have been processed. – scottmains Oct 21 '20 at 21:17
  • If you’re receiving a request via any methods you would send success after all records are processed, for API Gateway it must return the response in the correct format. If you must use the same Lambda you will need to format accordingly. Remember from API Gateway this will be a single HTTP request, not a batch – Chris Williams Oct 21 '20 at 21:31
  • Ok. I see a clear [format](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html) for API Gateway responses. I am looking for a format for a response to an SQS message with multiple records in the batch, or SES for example. This is what I meant by the question: How do I form my response payload(s) from a multiple-record event batch? I could return 'Success' as a string value; however, each record may require a specific response say some data or some calculation. If any of the records cause a handled exception, must the entire batch fail? – scottmains Oct 21 '20 at 21:55
  • You would send a success for the entire batch, generally you would only fail it in the event of a failure to perform the next step i.e. the Database is down. If individual records failed you would need to decide how to handle these within the code (perhaps manually adding to a DLQ) – Chris Williams Oct 22 '20 at 06:19
  • So in the case of a queue, there is not really a use case where the Lambda would respond with a list of responses for each individual record in the batch. Rather you return a generic 'Success' string. In that case why return anything at all? – scottmains Oct 22 '20 at 14:45
  • It's not required for certain services, it will be personal preference if you want to return a string on success. The response will be logged to logs so it may at least assist with your debugging – Chris Williams Oct 22 '20 at 14:55
0

A Lambda is invoked once per event whether the event contains a single request, or a batch of multiple records as with SQS Queue invocations. Using AWS Lambda with Amazon SQS or as Alex DeBrie of Serverless Blog writes: "With the SQS / Lambda integration, a batch of messages succeeds or fails together."Using SQS with AWS Lambda and Serverless

You can code your Lambda function to process each record of a batch in a loop and perform some action for each. After all the records are processed, the function may return a response payload. This could be the string "Success" or another message or data or None. How you format the payload, and what data you return depends on what needs to be done with it. Alternatively, the Lambda function can raise an error, or exit without returning anything; however, in these instances, the messages in the batch will remain visible in the queue.

As far as I can tell, what you return in your response payload doesn't mean anything to SQS. Success or failure determines whether a retry is necessary or whether the batch of messages can be deleted or moved to a dead letter queue. You may find it useful to return a meaningful message or other data since this will be logged; however, SQS does not do anything with it.

If individual messages in a batch inform downstream processes, your Lambda code can transmit individual responses to another service, such as another SQS queue or an API, one at a time as the records are processed.

scottmains
  • 31
  • 5