2

I am running an AWS Lambda function for image/video processing with Node 4.3 as a runtime. I am invoking my function from my webapp code with the node aws-sdk.

The issue is that when the function takes a long time to execute (e.g. 250 seconds), the invocation callback is never received, although I can clearly see in the aws cloudwatch logs that the function executed properly. On top of that the function is re-run again at least twice (this is probably related to the maxRetries parameter from the invocation: since it doesn't get a response back it retries).

What confuses me is that it works for quicker functions and my lambda function never times out. Anyone having such an issue ? Can it be related to the new 4.3 runtime? Note that I omit the context.succeed()or context.fail() as it is not required anymore, but I tried with it and it doesn't change a thing.

Lambda code

...
var handler = function (event, context, callback) {
    // video/image processing code
    // 
    // callback function
    ..., function(err, result) {
        if (err) {
            console.log("Error", err, err.stack);
            callback(err);
        } else {
            console.log("Done");
            callback(null, result);
        }
    }
};

Lambda Invocation

var lambda = new AWS.Lambda;

var myEventObject = {...};
var payload = JSON.stringify('myEventObject');

var params = {
    FunctionName: 'myLambdaFunction'
    InvocationType: 'RequestResponse',
    LogType: 'None',
    Payload: payload
};

lambda.invoke(params, function(err, data) {
    if (err) console.log(err, err.stack); // an error occurred
    else     console.log(data);           // successful response
});

Lambda CloudWatch Log

REPORT RequestId: xxx   Duration: 206757.82 ms  Billed Duration: 206800 ms Memory Size: 1536 MB Max Memory Used: 354 MB
Daniel T
  • 73
  • 2
  • 8
  • I suspect that the timeout config for the sdk is 240 seconds, which could explain that you get a retry. Could you try to change this config within you cli? – Tom Apr 15 '16 at 13:28
  • No the timeout config is set at max: 300 seconds. If it goes beyond 300 seconds the functions times out and I can see it in the cloudwatch logs. If I set the timeout to a lower value, say 30 seconds, the function returns with a timeout error and I catch the callback error. – Daniel T Apr 15 '16 at 17:20
  • 2
    please note that i m not talking about the timeout of lambda, but the one from the sdk config, so do you? – Tom Apr 15 '16 at 17:31
  • Oh sorry, my bad. I indeed missed this setting when looking at the aws doc. It works by adding the appropriate option which default at 120000 : `var lambda = new AWS.Lambda({httpOptions{timeout: 300000}});` Thank you so much – Daniel T Apr 16 '16 at 08:12
  • happy to help :) putting this comment as an answer for potential future readers – Tom Apr 16 '16 at 10:21

1 Answers1

5

as stated within the comments, the aws sdk has its own timeout config setting, that is by default set to 120 seconds (this is a different timeout setting than the lambda one).

in the current case, as the lambda runs for more this timeout, the sdk is assuming a failure and firing retries.

setting this config to 300 seconds should solve the problem. As Daniel T showed in the comment, temporary changing this can be achieved this way: var lambda = new AWS.Lambda({httpOptions{timeout: 300000}});

Tom
  • 2,689
  • 16
  • 21
  • How many retries does it by default? Unless success? This case looks very similar to what occurs for me here https://stackoverflow.com/questions/36648377/aws-lambda-invoke-function-doesnt-always-return I have already set timeout, but maybe it should be longer. The worst is that there is no info logged from aws-sdk that timeout happened and it tries invoking again. – Przemek Lewandowski May 18 '20 at 09:17
  • I think you can configure it in ClientConfiguration (attribute maxErrorRetry). see https://docs.aws.amazon.com/general/latest/gr/api-retries.html – Tom May 18 '20 at 09:48
  • to debug you an also execute the lambda locally, launching it locally actually exposes it through an RPC server. Or you can use sam which is probably calling via rpc as well https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-invoke.html – Tom May 18 '20 at 09:53
  • The problem is that this lambda im writing about is connecting with external source which have our aws env whitelisted, and our local envs not. I've just redeployed invoking service along with lambda - changed the timeout to be minute longer than possible lambda's timeout. Result is lambda does what it should, even gets to the return keyword, but the invoking service doesn't receive response with payload we need, instead it still timeouts despite lambda logging that it returnes value. Earlier it would trigger lambda again, but i have used that maxEntries set to 0. – Przemek Lewandowski May 18 '20 at 11:37
  • Do you have a link to a question? the one you pasted in your initial comment is actually this page – Tom May 18 '20 at 15:38
  • Oh, my bad. Will just post an update to it with what I came to yesterday https://stackoverflow.com/questions/61729392/aws-lambda-retries-despite-returning-value – Przemek Lewandowski May 19 '20 at 06:57