1

How to get of consumer.id which are listening to a topic using Confluent_Kafka?

I am able to get the consumer-groups and topic. But not the consumer name/id.

James Z
  • 12,209
  • 10
  • 24
  • 44
Marshall
  • 127
  • 3
  • 11
  • Basically I am looking for whether a consumer is running or not. If not running then I need to start it. Thank You – Marshall Aug 26 '21 at 06:39

1 Answers1

2

If your consumers are using group functionality, you can get a list of the consumer groups using the AdminClient and then find each consumer that is a member of the group. it has been added in this commit.

Example snippet

from confluent_kafka.admin import AdminClient
broker = '1.1.1.1:9092' # replace appropriately
a = AdminClient({'bootstrap.servers': broker})
groups = a.list_groups(timeout=10)
print(" {} consumer groups".format(len(groups)))
for g in groups:
    print(" \"{}\" with {} member(s), protocol: {}, protocol_type: {}".format(
        g, len(g.members), g.protocol, g.protocol_type))

    for m in g.members:
        print("id {} client_id: {} client_host: {}".format(m.id, m.client_id, m.client_host))

You may be interested in the client_id field that would differentiate the consumers in a group.

Reference: https://github.com/confluentinc/confluent-kafka-python/blob/master/examples/adminapi.py

JavaTechnical
  • 8,846
  • 8
  • 61
  • 97
  • 1
    Two Words. Awesome and Thank You – Marshall Aug 28 '21 at 13:10
  • Some times even though my consumer is running it is returning zero members from group. why that behavior ? Some time there will be no message to consumer for a long time, is that causing it ? – Marshall Sep 02 '21 at 18:13
  • 1
    @Marshall The consumer would likely have been kicked-out of the group, perhaps because it has failed to send heartbeats to Kafka. Enable DEBUG logs for kafka and check the issue. Also, double-check if the group you are looking is same as that of the consumer. – JavaTechnical Sep 03 '21 at 06:43
  • 1
    Additionally, you might also want to increase your `session.timeout.ms` to higher as applicable. – JavaTechnical Sep 03 '21 at 06:47