AWS Lambda retry using SQS DLQ

Question

What example belong does is, a cronjob runs every minute to trigger a lambda function written in Golang. If the lambda returns an error, a message is put into a DLQ straight away. However, what I am struggling to work out is that the retry logic. A message should go to DLQ only after third lambda try which is what I am trying to accomplish. If you see I am missing something in AWS commands below please let me know.

What I tried so far is that, I created an additional normal queue on top of DQL and linked it to lambda instead with --dead-letter-config. Then linked DLQ to target with DeadLetterConfig with RetryPolicy. I am not sure if this is how the whole thing is designed to work but I think there may be more components required for this. Not even sure if this is correct either!

Lambda (main.go)

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func main() {
    lambda.Start(handle)
}

func handle(_ context.Context, event events.CloudWatchEvent) error {
    detail, err := event.Detail.MarshalJSON()
    if err != nil {
        return err
    }

    if string(detail) == `{"ok": "yes"}` {
        return nil
    }

    return fmt.Errorf("not ok")
}

AWS steps

GOOS=linux CGO_ENABLED=0 go build -ldflags "-s -w" -o main main.go
zip main.zip main

# Create rule
aws --profile localstack --endpoint-url http://localhost:4566 events put-rule \
    --name test-rule \
    --schedule-expression 'cron(* * * * *)'

# Create DLQ
aws --profile localstack --endpoint-url http://localhost:4566 sqs create-queue \
    --queue-name test-dead-letter-queue \
    --attributes '{}'

# Create lambda with DLQ
aws --profile localstack --endpoint-url http://localhost:4566 lambda create-function \
    --function-name test-lambda \
    --handler main \
    --runtime go1.x \
    --role test-role \
    --dead-letter-config '{"TargetArn":"arn:aws:sqs:eu-west-1:000000000000:test-dead-letter-queue"}' \
    --zip-file fileb://main.zip

# Create lambda rule (purposely causes lambda error!)
aws --profile localstack --endpoint-url http://localhost:4566 events put-targets \
    --rule test-rule \
    --targets '[{"Id":"1","Arn":"arn:aws:lambda:eu-west-1:000000000000:function:test-lambda","Input":"{\"ok\":\"no\"}"}]'

I am not seeing what AWS doc says happening.

Error handling for a given event source depends on how Lambda is invoked. Amazon CloudWatch Events is configured to invoke a Lambda function asynchronously.

Asynchronous invocation – Asynchronous events are queued before being used to invoke the Lambda function. If AWS Lambda is unable to fully process the event, it will automatically retry the invocation twice, with delays between retries.

Answer 1

You have a asynchronous schedule rule -> lambda setup. Note that async retries are made with a delay (see below). Perhaps you are not noticing the retries because the backoff time >= your per-minute schedule?

Lambda error handling and automatic retry logic is different for async and sync invocation patterns:

Asynchronous (e.g. event-triggered) Lambda Invocations

Two async lambda retries are made with a 1 minute and 2 minute backoff, respectively. Async lambdas will be retried in the case of both invocation and function errors. After the retries are exhausted, the event will be sent to the lambda's DLQ or the lambda's failure destination^*, if configured.

Synchronous (e.g. SQS-triggered) Lambda Invocations

An alternative is to use SQS as an event source (schedule rule -> Queue -> Lambda). In this synchronous scenario, retry logic and the DLQ is configured on the Queue itself, not on the Lambda.

---

* Destinations are a newer alternative to async lambda DLQs: "Destinations and DLQs can be used together and at the same time although Destinations should be considered a more preferred solution."