0

I have a Kinesis consumer written in Java with KCL. From the consumer side, I can throttle the rate with setMaxRecords method to set the maximum limit of how many records within one batch.

My question is: if I don't set any limit, what is the logic to determine how many records in one batch? Is there a default value for the max allowed?

ethan
  • 1,881
  • 2
  • 17
  • 31

1 Answers1

0

When using KCL, the default maximum for number of records in a batch is 10000.

See the KinesisClientLibConfiguration source code for more details.

Krease
  • 15,805
  • 8
  • 54
  • 86
  • Thanks. The max defaults to 10k. In my case, I have 100k in one steam one shard, I start consumer and then monitor the records for each batch. The number of records for each batch varies, and never reaches 10k. Where can I find the logic for KCL to determine how many records to pull? – ethan Jun 07 '18 at 15:48
  • The number of records returned is limited not just by maxRecords, but also by record size (max 10MB per call), throughput used (2MB per second per shard), See the [getRecords documentation](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetRecords.html) and [limitations documentation](https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html) – Krease Jun 07 '18 at 16:54
  • So do you imply that the KCL will dynamically calculate the number of records to pull based on the load? Do you happen to know the source code file that contain this logic? In this case, if KCL calculate the file size, will "rate exceeded" exception still happen? – ethan Jun 07 '18 at 17:28
  • It isn't KCL that does that, it's the backend throttling mechanism of Kinesis. – Krease Jun 08 '18 at 16:58