16

My node4 lambda function called via API GW makes a sequence of slow API calls. In order to not let users wait until everything completes, I'm planning to have my code look like this:

function(event, context, callback) {
  ...
  // Return users API GW call now
  callback(null, data);
  // Do the heavy lifting afterwards.
  longApiCall().then(otherLongApiCalls)
}

But now I read in the AWS docs: "the callback will wait until the Node.js runtime event loop is empty before freezing the process and returning the results to the caller"

Does that mean the API GW returns the response data before or after the longApiCalls complete?

If after, is there a suggested way for how to "return early" before everything is finished?

bebbi
  • 2,489
  • 3
  • 22
  • 36

2 Answers2

26

In your current configuration API Gateway will wait until the Lambda function has finished executing before sending a response. Your options are:

  1. Change the API Gateway endpoint's integration type to AWS Service and have API Gateway invoke the Lambda function asynchronously. This is documented here.
  2. Have the Lambda function that API Gateway invokes do nothing but invoke another Lambda function asynchronously and then return.
  3. Have API Gateway, or a Lambda function called by API Gateway, send a message to an SNS topic. Then have the SNS topic trigger a Lambda function that handles the long API calls. This would decouple your microservices a bit.
  4. Have API Gateway, or a Lambda function called by API Gateway, trigger an AWS Step Function that is configured to handle the long API calls via one or multiple Lambda functions. I would suggest this approach if the long API calls run the risk of running over a single Lambda function's execution time limit of 5 minutes.
Mark B
  • 183,023
  • 24
  • 297
  • 295
  • 1
    FWIW, I went along #2 and got bitten by a difference of limits: While sync lambda calls handle a 6MB body, a lambda run as an event errors on a body >128K! I assume most or all solutions will suffer this issue. http://docs.aws.amazon.com/lambda/latest/dg/limits.html – bebbi Mar 28 '17 at 19:44
  • 1
    Note for #1 you cannot return data from lambda as per linked document. – ian May 16 '18 at 05:28
2

Option 5. Let your lambda function queue a message to SQS and poll the queue from another lambda or ec2 or wherer you want to do the heavy lifting.

Andi
  • 404
  • 5
  • 7
  • 1
    You cannot poll an SQS from another lambda. In order to do that you need to trigger the lambda via a cron and that needs to then check if SQS has a message. If in that run you didnt get a message you need to wait until the cron triggers the lambda again. You are better of with an EC2 in that case. – sethu Jul 04 '17 at 23:00
  • You can easily setup a Cloudwatch Event that triggers the lambda every minute. Or since last year you can just have the Lambda invoked by SQS: https://aws.amazon.com/de/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/ – Andi Jul 10 '19 at 22:43