8

I'm in the process of writing a Lambda function that processes items from a DynamoDB stream.

I thought part of the point behind Lambda was that if I have a large burst of events, it'll spin up enough instances to get through them concurrently, rather than feeding them sequentially through a single instance. As long as two events have a different key, I am fine with them being processed out of order.

However, I just read this page on Understanding Retry Behavior, which says:

For stream-based event sources (Amazon Kinesis Data Streams and DynamoDB streams), AWS Lambda polls your stream and invokes your Lambda function. Therefore, if a Lambda function fails, AWS Lambda attempts to process the erring batch of records until the time the data expires, which can be up to seven days for Amazon Kinesis Data Streams. The exception is treated as blocking, and AWS Lambda will not read any new records from the stream until the failed batch of records either expires or processed successfully. This ensures that AWS Lambda processes the stream events in order.

Does "AWS Lambda processes the stream events in order" mean Lambda cannot process multiple events concurrently? Is there any way to have it process events from distinct keys concurrently?

Laurence Gonsalves
  • 137,896
  • 35
  • 246
  • 299

2 Answers2

7

With AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sources, the order is still guaranteed for each partition key, but not necessarily within each shard when Concurrent batches per shard is set to be greater than 1. Therefore the accepted answer needs to be revised.

Treefish Zhang
  • 1,131
  • 1
  • 15
  • 27
  • "Parallelization factor" and "Concurrent batches" mentioned in the doc linked mean the same thing. Parallelization factor appears in APIs and the console shows the parameter as "Concurrent batches per shard". – nonylene Apr 13 '23 at 09:34
4

Stream records are organized into groups, or shards.

According to Lambda documentation, the concurrency is achieved on shard-level. Within each shard, the stream events are processed in order.

Stream-based event sources : for Lambda functions that process Kinesis or DynamoDB streams the number of shards is the unit of concurrency. If your stream has 100 active shards, there will be at most 100 Lambda function invocations running concurrently. This is because Lambda processes each shard’s events in sequence.

And according to Limits in DynamoDB,

Do not allow more than two processes to read from the same DynamoDB Streams shard at the same time. Exceeding this limit can result in request throttling.

codigube
  • 1,186
  • 1
  • 12
  • 31