2

I understand that when a Lambda times-out, it returns an error.

Once the Lambda times-out, though, I'd like to know if it's immediately killed by AWS, or if it can potentially continue running for an arbitrary amount of time in the background (detached from the original request, so to speak).

I ask because it's important in my situation that when a Lambda times out, all processing inside that Lambda also stops immediately. If AWS does not guarantee this, then I will need to implement my own measures to ensure this for myself, i.e. by periodically checking against the clock and manually halting execution beyond the timeout.

(p.s. if Lambdas are killed immediately on timeout, I'd like to know how AWS achieves keeping the Lambda in a warm, initialised state for subsequent invocations. I.e. it cannot simply be that it kills the process on timeout, otherwise the Lambda would be cold for next time...)

Lawrence Wagerfield
  • 6,471
  • 5
  • 42
  • 84

2 Answers2

5

The execution of the lambda function will stop, but not necesarily the execution of external processes that have been called from lambda, such as external API or other AWS services.

To avoid issues, lambda has a context method getRemainingTimeInMillis(), allowing to check during execution how much time is left before the timeout. https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html

From my own experience, most of the time the lambda stays warm. AWS killed the execution of the code, not the instance.

Jeremie
  • 646
  • 8
  • 24
  • 2
    Just a note to the answer - there is an execution leak using nodejs async functions, where unfinished async may continue when reusing the execution context https://lumigo.io/blog/node-js-lambda-execution-leaks-a-practical-guide/ – gusto2 Nov 26 '20 at 21:20
  • 1
    "AWS killed the execution of the code, not the instance." -- I'm curious to know how they do it. How can you "kill" a function? Is it as simple as clearing-down the event queue + any outstanding interrupts/eventHandlers to ensure "nothing else executes"? – Lawrence Wagerfield Nov 26 '20 at 21:22
  • 2
    I don't know how they do it. What I found out is that the lambda stayed warm, so I guess the instance/container was not killed – Jeremie Nov 26 '20 at 21:25
3

From what you've posted I think you will need some sort of orchestration than relying on the behavior characteristics of the AWS service. Rather develop using the matching service to meet your requirements.

This may mean instead of calling vanilla Lambda, you use a StepFunction to handle the Lambda timeout, clean-up on fail, and retry logic. Or maybe you need an actual lock with a TTL and you may use DynamoDB for this with a TTL attribute enabled, it will all depend on your requirements.

In general I don't think you can rely that Lambda, or any service (rather than an ACID transaction with in a database) will execute in a strongly consistent manor which you can rely on. If your requirements are not too strict, you may get away with simply delaying the invocation of the next attempt. This could be done with an SQS delay queues. FIFO can help you with preventing duplicates as well if you decide to use SQS.

Derrops
  • 7,651
  • 5
  • 30
  • 60
  • 1
    Step Functions are not a solution for any workflow requiring significant volume, due to the cost of Step Functions ON TOP of lambda function cost: 1) Standard Workflows cost 25 USD / 1000000 transitions (125x more expensive per execution than standard lambda, not considering GB-hours) ; 2) Express Workflows cost 1 USD / 1000000 requests + GB-hours (when your main lambda is executing you also pay for step function waiting!). In these case, integrating the timeout check inside lambda using getRemainingTimeInMillis() is less expensive. – Jeremie Dec 03 '20 at 14:49
  • Same with DynamoDB, which will increase cost with read/write requests and may generate problems if not strongly consistent. – Jeremie Dec 03 '20 at 14:52
  • You are quiet right, costs need to be taken into consideration if your volumes are very high. We process 10's of thousands of files with step funcs and its actually cheaper though because there is a great deal of waiting time for other processes, so you need to figure out if you do have this requirement. But if do require consistency you will most likely need some orchestration tech, or your downstream system needs to handle it. – Derrops Dec 04 '20 at 06:47