1

I have a Lambda which is configured as a consumer of a Kinesis data stream, with a batch size of 10,000 (maximal). The lambda parses given records and inserts them to Aurora Postgresql (using an INSERT command).

Somehow, I see that the lambda is invoked most of the time with a relatively small number of records (less than 200), although the 'IteratorAge' is constantly high (about 60 seconds). The records are put in the stream with a random partition key (generated as uuid4), and of size

How can that be explained? As I understand, if the shard is not empty, all the current records, up to the configured batch size, should be polled.

I assume that if the Lambda was invoked with bigger batches this delay could be prevented.

Note: There is also a Kinesis Firehose configured as a consumer (doesn't seem to have any issue).

Ronyis
  • 1,803
  • 16
  • 17

1 Answers1

0

Finds out that the iterator age of the Kinesis was 0ms, so this behavior makes sense. The iterator age of the Lambda is a bit different:

Measures the age of the last record for each batch of records processed. Age is the difference between the time Lambda received the batch, and the time the last record in the batch was written to the stream.

Ronyis
  • 1,803
  • 16
  • 17