how can I prevent dynamoDB Stream handler from infinitely processing a record when I use batchItemFailures

Question

I have a dynamoDB stream which is triggering a lambda handler that looks like this:

let failedRequestId: string
await asyncForEachSerial(event.Records, async (record) => {
    try {
        await handle(record.dynamodb.OldImage, record.dynamodb.NewImage, record, context)
        return true
    } catch (e) {
        failedRequestId = record.dynamodb.SequenceNumber
    }
    return false //break;
})

return {
    batchItemFailures:[ { itemIdentifier: failedRequestId } ]
}

I have my lambda set up with a DestinationConfig.onFailure pointing to a DLQ I configured in SQS. The idea behind the handler is to process a batch of events and interrupt at the first failure. Then it reports the most recent failure in 'batchItemFailures' which tells the stream to continue at that record next try. (I pulled the idea from this article)

My current issue is that if there is a genuine failure of my handle() function on one of those records, then my exit code will trigger that record as my checkpoint for the next handler call. However the dlq condition doesn't ever trigger and I end up processing that record over and over again. I should also note that I am trying to avoid reprocessing records multiple times since handle() is not idempotent.

How can I elegantly handle errors while maintaining batching, but without triggering my handle() function more than once for well-behaved stream records?

Answer 1

I'm not sure if you have found the answer you were looking for. I'll respond in case someone else come across this issue.

There are 2 other parameters you'd want to use to avoid that issue. Quoting documentation (https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html):

Retry attempts – The maximum number of times that Lambda retries when the function returns an error. This doesn't apply to service errors or throttles where the batch didn't reach the function.

Maximum age of record – The maximum age of a record that Lambda sends to your function.

Basically, you'll have to specify how many time the failures should be retried and how far back in the events Lambda should be looking at.