2

What is the batching window used for MSK as an event source? It seems there is a support for Batch Window and MaximumBatchingWindowInSeconds for SQS as an event source, however how to provide the same params for lambda for MSK source

https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

Does it mean that MSK will wait indefinitely to fill the BatchSize ?

vin
  • 960
  • 2
  • 14
  • 28

1 Answers1

1

Yes, there is an option to set Batch Window for MSK trigger. When you create the trigger you can set the Batch Window as well as Batch Size on lambda console it self. Internally it's a event source mapping which polls the messages from the topic with these settings. Imagine a internal consumer consuming these messages and once the threshold is reached, another process invokes the lambda function. The threshold here can be defined as:

  1. Either the set Batch size is reached.
  2. Or the Batch Window is reached.
  3. Or the size of the events gathered reached 6MB. In this case the last message is dropped and polled in next batch.

PS: Event polling by event source mapping is a synchronous process. As long as the lambda does not exit the execution successfully, the entire batch offset is not committed. It blocks the execution on that particular partition for this duration.

From AWS Docs- https://aws.amazon.com/blogs/compute/introducing-aws-lambda-batching-controls-for-message-broker-services/

Mradul Yd
  • 71
  • 8