0

I want to get the details of the consumer group using confluent-kafka. The cli equivalent of that is `

./kafka-consumer-groups.sh --bootstrap-server XXXXXXXXX:9092 --describe --group my-group

My end goal is to get the value of lag from the output. Is there any method in confluent-kafka python API to get these details. There is a method in the java API but I couldn't find it in python API.

I tried using the describe_configs method in the adminClient API but it ended up throwing kafkaException with following details

This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Taimoor Abbasi
  • 51
  • 6

1 Answers1

0

For now I have come up with the following solution. It's a work around to get the combined lag of a consumer group

 def get_lag(topic,numPartitions):
    diff = list()
    for i in range(numPartitions):
        topic_partition = TopicPartition(topic, partition=i)
        low, high = consumer.get_watermark_offsets(topic_partition)
        currentList = consumer.committed([topic_partition])
        current = currentList[0].offset
        diff.append(high-current)
    return sum(diff) # Combined Lag
Taimoor Abbasi
  • 51
  • 6
  • It's not a workaround. Using the consumer class is the only way – OneCricketeer Nov 25 '22 at 00:39
  • 1
    I'd use consumer.list_topic to get metadata for finding partitions, instead of passing numPartitions. Also you have to check high < 0 to ignore such partition, also check if .offset < 0 for high - low as lag. Personally, I added admin_client to get host name where (topic, partition) consumers are running to identify which host is having problem. – Kenji Noguchi May 17 '23 at 20:56