0

is there tool available to measure Kafka re-balancing duration? or check any intermediate status?

we have observe a many time, specific consumer get stuck forever during Kafka rebalancing, we never waited to finish.

kus
  • 446
  • 3
  • 7

1 Answers1

0

There was a bug in Apache Kafka 2.4.1 which got fixed, and MSK has 2.4.1.1. This was exactly related to infinite rebalance of consumers.

If your MSK running later versions, you can open support case.

EdbE
  • 204
  • 1
  • 4
  • is there tool available or statistics within Kafka to know if if Kafka ReBalance occured? – kus Jan 16 '22 at 03:34
  • in MSK, broker logs will have rebalance trace. you can analyze the logs by delivering them to Splunk, Elastic/Open-Search, etc... or even pushing to firehose with lambda. – EdbE Jan 20 '22 at 02:45