How to create topics if it does not exists in Kafka dynamically using kafka-python

Question

I'm fairly new to Python and just getting started with Kafka. I am using the library named python-kafka to communicate with Kafka. Now I have a requirement that I need to create topics dynamically, however if it does exists, I do not need to create it.

From reading the docs I see that I can use KafkaAdminClient to create and delete topics, however I do not find any to check if topic exists.

score 5 · Accepted Answer · answered Apr 03 '19 at 11:42

5

The KafkaAdminClient does not expose a method to list topics but you can get the list of existing topics by simply querying the cluster metadata from a KafkaClient.

For example, this will print all the topics in the cluster:

from kafka.client import KafkaClient

client = KafkaClient(bootstrap_servers='localhost:9092')

future = client.cluster.request_update()
client.poll(future=future)

metadata = client.cluster
print(metadata.topics())

answered Apr 03 '19 at 11:42

Mickael Maison

25,067
7
71
68

1

For the newer versions use ```from kafka import KafkaClien``` as the project structure is changed. – Arian Maghsoudnia Dec 16 '21 at 13:57

How to create topics if it does not exists in Kafka dynamically using kafka-python

1 Answers1