0

We have a java based KCL worker consuming records from DynamoDB streams. I want to understand based on high number of partition in table and shards in streams, what values should be given while creating worker. Also these tables are actively merging and splitting shards. Asking about following properties (with default values) -

failoverTimeMillis = 10000
shardSyncIntervalMillis = 60000
Mahesh Nagarwal
  • 55
  • 1
  • 1
  • 6
  • Tables never merge partitions. Stream shards roll-over every 4 hours. – Leeroy Hannigan Feb 22 '23 at 11:36
  • Sorry, my bad. But is there any mathematical explanation to explain how long a lease should be or how often sync should happen based on high number of active shards. I don't think on high scale I should rely on default configuration. I can already see limit_exceed_exception when shards are synced due to calling describe_stream internally. – Mahesh Nagarwal Feb 22 '23 at 11:41

0 Answers0