3

I have an AWS API Gateway method that proxies requests through to AWS Lambda. However, it errors after three seconds with the following in the logs:

Endpoint response body before transformations: {"errorMessage":"2017-09-05T16:30:49.987Z 922186c0-9257-11e7-9db3-51921d5597a2 Task timed out after 3.00 seconds"}

Thus, I went on to check my Node 6.10 AWS Lambda function to see why it was timing out. I added logging statements before and after every function call. Surprisingly, it did everything it's supposed to do: called the API Gateway callback, and run a query against the database after that. All that takes 0.6s, and as far as I'm aware there's no other code left to run. Nevertheless, it appears to keep on running for the rest of the three seconds and then timing out. (This is, I think, because I'm leaving a connection to the database open.)

The logs statements I placed before and after the callback call indicate that the that call is executed in under half a second. Yet, that response doesn't seem to make it to API Gateway, whereas the error after three seconds does.

What could be potential reasons for this, and how can I debug it?

Vincent
  • 4,876
  • 3
  • 44
  • 55

3 Answers3

13

By default calling the callback() function in a NodeJS Lambda function does not end the function execution. It will continue running until the event loop is empty. A common issue with NodeJS Lambda functions continuing to run after callback is called occurs when you are holding on to open database connections. You haven't posted any code, so I can't give specific recommendations, but you would need to determine if you are leaving database connections open in your code or something similar.

Alternatively, you can change the behavior such that the execution ends as soon as the callback function is called by setting callbackWaitsForEmptyEventLoop = false on the context object.

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • Hmm yeah, talking about it made me think in that direction as well. I'm actually leaving the database connection open as I'd understood that it could then be re-used if the Lambda container happened to be re-used, but I'm starting to understand that that doesn't apply when using API Gateway. I actually do want execution to continue, but I guess can't really respond to API Gateway meanwhile. I'm looking into closing the database connection now, but that's probably it. – Vincent Sep 05 '17 at 20:00
  • Default is `true`... you'd want to set `callbackWaitsForEmptyEventLoop` = `false`. – Michael - sqlbot Sep 06 '17 at 00:34
  • 1
    @Michael-sqlbot right. Thanks for pointing that out. Fixed in answer. – Mark B Sep 06 '17 at 00:43
2

Your API gateway has a fixed timeout of 29 seconds. Most queries do complete within this timeframe. Increase your lambda execution timeout to anywhere between 30 Sec to 3 Min 00 Sec. Use context.succeed() instead of callback. This worked for me.

const mysql = require('mysql');
const connection = mysql.createConnection({
  host     : 'your_mysql_host',
  user     : 'your_mysql_user',
  password : 'your_mysql_password',
  database : 'your_mysql_db'
});

exports.handler = (event, context) => {
  var userId = event.params.querystring.userid;
  const sql = 'SELECT * FROM users where USER_ID=' + userId;

  var response = {
    "statusCode": 200, 
    "body": "body_text_goes_here"
  }

  if(userId){
    connection.query(sql, function (error, results, fields) {
      if (error) {
        context.succeed(error);
      } else {
        response.body = results;
        context.succeed(response);
      }
    });
  }
}
Srirag Nair
  • 166
  • 1
  • 8
  • 1
    [The documentation](https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html) seems to imply that `context.succeed` is a remnant of earlier runtimes (Node 0.10) and that, just like the `callback` parameter, it doesn't result in early termination of the Lambda function (in your code example, the function only terminates after `context.succeed` because it is the last function call in the handler). – Vincent Jan 10 '18 at 09:59
  • Though it appears to be the last function call, the program however is not terminated as the mysql connection is still open. Due to this, the callback return data is not treated as a response (in the case of API Gateway usage). This case is particularly in the scenarios where lambda's are used in conjunction with API Gateways. However, I have found in many of my lambda usages across many functions that using context.succeed tended to inform my api gateway that the results returned are what its waiting for. – Srirag Nair Jan 17 '18 at 10:19
-2

I had same issue and I've updated the timeout in my code but no luck finally increased lambda execution time that fixed my problem.

How to increase an AWS Lambda timeout?

Vasanth Umapathy
  • 1,571
  • 14
  • 7