2

I have a Lambda function that is set to timeout at 5 minutes, and occasionally when I invoke the function using the node SDK, I am getting a timeout error after 5 minutes.

However, when I have checked the CloudWatch logs after getting the client-side error, and am seeing the function successfully complete after ~3:30 minutes of execution time.

I have also noticed that the client side will log the timeout 2 minutes before the CloudWatch logs read that the function completed (ie, client side error timestamps at 4:08 and the CloudWatch successful end timestamps at 4:10), but I am assuming that is a lag between log writes, not run time. Thought I would mention it just in case that is something worth thinking more about.

From what I can tell, when the function runs under two minutes, it is consistently getting a positive response.

Code looks like this:

let aws_config = new AWS.Config({
    region: 'us-east-1',
    credentials: credentials,
    httpOptions: { 
        proxy: process.env.HTTP_PROXY,
        timeout: 300000
    }
})

var lambda = new AWS.Lambda(aws_config);

lambda.invoke(params, (err, data: Result) => {
  ...
}

Error looks like this:

{ TimeoutError: Connection timed out after 300000ms
at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)
at Object.onceWrapper (events.js:273:13)
at ClientRequest.emit (events.js:182:13)
at ClientRequest.EventEmitter.emit (domain.js:442:20)
at Socket.emitRequestTimeout (_http_client.js:661:40)
at Object.onceWrapper (events.js:273:13)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:442:20)
at Socket._onTimeout (net.js:449:8)
at ontimeout (timers.js:425:11)
message: 'Connection timed out after 300000ms',
  code: 'TimeoutError',
  time: 2018-10-05T20:08:35.719Z,
  region: 'us-east-1',
  hostname: 'lambda.us-east-1.amazonaws.com',
  retryable: true } 'TimeoutError: Connection timed out after 300000ms\n    at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)\n    at Object.onceWrapper (events.js:273:13)\n    at ClientRequest.emit (events.js:182:13)\n    at ClientRequest.EventEmitter.emit (domain.js:442:20)\n    at Socket.emitRequestTimeout (_http_client.js:661:40)\n    at Object.onceWrapper (events.js:273:13)\n    at Socket.emit (events.js:182:13)\n    at Socket.EventEmitter.emit (domain.js:442:20)\n    at Socket._onTimeout (net.js:449:8)\n    at ontimeout (timers.js:425:11)'
getglad
  • 2,514
  • 3
  • 24
  • 47
  • Did you try to increase Memory limit for your lambda function? – Nikolay Vetrov Oct 05 '18 at 21:01
  • I can try, but I am really thinking there is a SDK/client side problem, as I'm not seeing errors in the console. Also, max memory usage is under 100MB. – getglad Oct 05 '18 at 21:03
  • What's missing here is the Lambda function's code, and the function execution log. Most of the times I've seen this kind of behaviour was when the function did not terminate *correctly*, i.e., did not call the callback function but simply finished the run. – Kalev Oct 05 '18 at 21:05
  • @Kalev - function code is large. It returns most of the time and logs are showing successful return, not just successful run of the code. So I do think this is client side. – getglad Oct 05 '18 at 21:11
  • @getglad - when you run the function with the same parameters from the AWS web console what behaviour do you witness? Also, creating a minimal example that reproduces the behaviour would go a long way towards isolating the issue. – Kalev Oct 05 '18 at 21:14
  • @Kalev - it runs successful. Also do not have this problem using python SDK. Will work on getting your minimal example up. – getglad Oct 05 '18 at 21:19
  • @getglad As I am also have faced this issue in lambda. when I am using third party call and its connection still open so, lambda is waiting for terminating that event. So its good to you provide your code what exactly you use – IftekharDani Oct 06 '18 at 07:56

1 Answers1

1

Have you looked into 

context.callbackWaitsForEmptyEventLoop = false;

Here'e a related SO posting: Why does AWS Lambda function always time out?

Marc
  • 174
  • 4
  • `context.callbackWaitsForEmptyEventLoop = false;` this solve problem but its not good practice because when you `false` the `callbackWaitsForEmptyEventLoop ` than `AWS Lambda to freeze the process soon after the callback is called` – IftekharDani Oct 06 '18 at 07:58
  • callbackWaitsForEmptyEventLoop isn't bad practice and in fact in almost every production setting you'll have to use it for when you have open (My)SQL connection pools etc, it's just a fact of life on Lamdba unfortunately. That being said if the function has something you CAN close then do it rather than using the solution above. – Mrk Fldig Oct 06 '18 at 10:03