3

I'm looking into serverless technology (specifically, Python, Django and Zappa on AWS Lambda) and one thing about error handling struck me. In the Zappa docs it says

By default, AWS Lambda will attempt to retry an event based (non-API Gateway, e.g. CloudWatch) invocation if an exception has been thrown.

In the AWS Lambda documentation, I read:

Depending on the event source, AWS Lambda may retry the failed Lambda function. For example, if Kinesis is the event source, AWS Lambda will retry the failed invocation until the Lambda function succeeds or the records in the stream expire.

Does this mean a function will be called an infinite number of times when it raises an unhandled exception? If this goes on unchecked, the costs must go through the roof.

Related to that; what is meant by "until the records in the stream expire"? What records, and what stream?

Zanon
  • 29,231
  • 20
  • 113
  • 126
Mathieu Dhondt
  • 8,405
  • 5
  • 37
  • 58

1 Answers1

2

According with AWS docs:

  • Event sources that aren't stream-based: like S3, API Gateway, etc.

    • Synchronous invocation: if you have invoked the Lambda using the SDK or API Gateway, if an exception occurs, you will be responsible to decide if/when/how the request should be retried.

    • Asynchronous invocation: if the Lambda was triggered through an async invocation (like S3), it will automatically retry the invocation twice, with delays between retries. If you have specified a Dead Letter Queue, then the failed event is sent to SQS/SNS. If DLQ was not specified, the event will be discarded.

  • Stream-based event sources: like DynamoDB and Kinesis.

    • If a Lambda function fails, it will continue to try until the data expires (max of 7 days for Kinesis). It retries following a exponential backoff with the ceiling of 1 minute between two retries. You will pay for all retries, but you can create an alert to trigger and stop the stream when the source is offline.

The documentation regarding the stream-based event source is not very accurated, but you can read this thread in AWS forums where an AWS employee has answered a question about this:

Question:

Specifically, when my Lambda is getting Kinesis events and writing the data to another service... but the other service goes down for a period of time (e.g., a few hours)... is my Lambda going to keep getting called (and throwing errors) at a constant rate?

Lambda retry is good because I want guaranteed delivery of events, but ideally in this situation, I also don't want to be billed at a high rate when my Lambda becomes consistently UNsuccessful for a time

Answer:

If the function starts executing but fails because of a downstream dependency, then you do get billed for the duration the function ran. Lambda exponentially backs off in case your function fails, up to about one minute. You can also monitor this as the ShardIteratorAge increases, and take action to pause your stream processing if needed till you resolve the downstream dependency

Community
  • 1
  • 1
Zanon
  • 29,231
  • 20
  • 113
  • 126
  • Thanks! Just to be perfectly clear: if an exception occurs after a *synchronous* invocation, the request is *not* retried by default? – Mathieu Dhondt Jul 26 '17 at 12:52
  • 1
    If it's a *synchronous* invocation, the request is **not** retried by default. I am sure that Lambda will not automatically retry because I've tested this a few days ago and the log messages shows just the ones that I have invoked myself. – Zanon Jul 26 '17 at 13:03