4

I'm beginner to kafka client in python, i need some help to describe the topics using the client.

I was able to list all my kafka topics using the following code:-

consumer = kafka.KafkaConsumer(group_id='test', bootstrap_servers=['kafka1'])
topicList = consumer.topics()
Tech at The Sparks Foundation
  • 1,123
  • 2
  • 7
  • 15
Ashwin
  • 439
  • 1
  • 7
  • 23

4 Answers4

12

After referring multiple articles and code samples, I was able to do this through describe_configs using confluent_kafka.

Link 1 [Confluent-kafka-python] Link 2 Git Sample

Below is my sample code!!

from confluent_kafka.admin import AdminClient, NewTopic, NewPartitions, ConfigResource
import confluent_kafka
import concurrent.futures

#Creation of config
conf = {'bootstrap.servers': 'kafka1','session.timeout.ms': 6000}
adminClient = AdminClient(conf)
topic_configResource = adminClient.describe_configs([ConfigResource(confluent_kafka.admin.RESOURCE_TOPIC, "myTopic")])
    for j in concurrent.futures.as_completed(iter(topic_configResource.values())):
        config_response = j.result(timeout=1)
Ashwin
  • 439
  • 1
  • 7
  • 23
7

I have found how to do it with kafka-python:

from kafka.admin import KafkaAdminClient, ConfigResource, ConfigResourceType
KAFKA_URL = "localhost:9092" # kafka broker
KAFKA_TOPIC = "test" # topic name

admin_client = KafkaAdminClient(bootstrap_servers=[KAFKA_URL])
configs = admin_client.describe_configs(config_resources=[ConfigResource(ConfigResourceType.TOPIC, KAFKA_TOPIC)])
config_list = configs.resources[0][4]

In config_list (list of tuples) you have all the configs for the topic.

Scott
  • 1,648
  • 13
  • 21
Alvaro del Castillo
  • 141
  • 1
  • 4
2

Refer: https://docs.confluent.io/current/clients/confluent-kafka-python/

  1. list_topics provide confluent_kafka.admin.TopicMetadata (topic, partitions)

  2. kafka.admin.TopicMetadata.partitions provide: confluent_kafka.admin.PartitionMetadata (Partition id, leader, replicas, isrs) 

    from confluent_kafka.admin import AdminClient
kafka_admin = AdminClient({"bootstrap.servers": bootstrap_servers})    
for topic in topics:    
    x = kafka_admin.list_topics(topic=topic)    
    print x.topics, '\n'    
    for key, value in x.topics.items():    
        for keyy, valuey in value.partitions.items():    
            print keyy, ' Partition id : ', valuey, 'leader : ', valuey.leader,' replica: ', valuey.replicas
Lav Kush
  • 21
  • 1
0

Interestingly, for Java this functionality (describeTopics()) sits within the KafkaAdminCLient.java.

So, I was trying to look for the python equivalent of the same and I discovered the code repository of kafka-python.

The documentation (in-line comments) in admin-client equivalent in kafka-python package says the following:

describe topics functionality is in ClusterMetadata
Note: if implemented here, send the request to the controller

I then switched to cluster.py file in the same repository. This contains the topics() function that you've used to retrieve the list of topics and the following 2 functions that could help you achieve the describe functionality:

  1. partitions_for_topic() - Return set of all partitions for topic (whether available or not)
  2. available_partitions_for_topic() - Return set of partitions with known leaders

Note: I haven't tried this myself so I'm not entierly sure if the behaviour would be identical to what you would see in the result for kafka-topics --describe ... command but worth a try.

I hope this helps!

Lalit
  • 1,944
  • 12
  • 20
  • Thanks for explanation Lalit. I looked at the available methods no luck. I was actually looking at getting the retention period per topic which we get through kafka-topics.sh --describe. – Ashwin May 23 '19 at 13:06
  • Okay. Fair enough. I'll try and find something but it seems like python libraries are slightly different and not exactly analogous. – Lalit May 23 '19 at 16:09