0

I would like to monitor the consumer groups (Kafka based groups not ZK) in my Kafka cluster from within a Python application (I am open to using either of the Kafka-python library and the confluent Kafka client). The purpose is to launch more (or delete some) consumers based on each Consumer-Group's lag.

The confluent Kafka admin API does not seem to have a Python option to do this. Burrow seems to be an option. But it will be an extra service to set up and monitor.

Any alternate approaches are also welcome. I am looking for the best practice to do this.

Thank you!

void
  • 2,403
  • 6
  • 28
  • 53

1 Answers1

2

If your Kafka clients and brokers are modern enough you can accomplish this by monitoring the __consumer-offsets topic, defining some level of lag threshold which causes you to scale your consumer groups. This would be out of scope for the AdminClient to handle, as it's related to how many consumers you're launching on the application layer, and really doesn't have anything to do with broker operations.

I would also ensure your Kafka topics have enough partitions to be able to elastically scale your consumer groups.

Chris Matta
  • 3,263
  • 3
  • 35
  • 48