0

I am new to the kafka world and trying to do the following for a kafka consumer in python

  1. get a list of all kafka topics.
  2. get a list of topics a consumer has subscribed to.
  3. subscribe to new topics (that have not yet been subscribed to).

Note: I am ok to use either confluent-kafka / kafka-python library to achieve this.

Any help would be appreciated.

rocky502
  • 23
  • 6

1 Answers1

1

if you created your Consumer with kafka-python

from kafka import KafkaConsumer

consumer = KafkaConsumer(
 bootstrap_servers = 'hostname:port',
)

you can review the list of topics available with

consumer.topics()

when you subscribe to topics, you can review the consumer subscriptions with

consumer.subscription()

You can do one minus the other to find the topics you still need to subscribe to and then you can do so with

consumer.subscribe(topics=[list_of_topic_names])
Ftisiot
  • 1,808
  • 1
  • 7
  • 13
  • Looks like subscribe() does not allow incremental topic subscription. The list provided in subscribe() fn will replace the existing subscription list. If so, is there any side effect for re-subscribing to the same topics? More info: https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html#kafka.KafkaConsumer.subscribe – rocky502 Apr 28 '21 at 16:27