Kafka-python get number of partitions for topic

Question

I'm using: https://github.com/mumrah/kafka-python as a kafka api in Python. I want to get the number of partitions for a specified topic. How do I do that?

score 14 · Answer 1 · answered May 18 '15 at 14:07

14

Might be a slightly simplistic solution, but:

from kafka import KafkaClient

client = KafkaClient('SERVER:PORT')
topic_partition_ids = client.get_partition_ids_for_topic(b'TOPIC')
len(topic_partition_ids)

tested on Python 3.4.3 / kafka-python 0.9.3

answered May 18 '15 at 14:07

Chaffelson

1,249
9
20

1

consider adding: a = len(topic_partition_ids) print(topic_partition_ids) print(a) – Rubber Duck Jul 05 '18 at 06:19
3

this has been deprecated, try using "partitions_for_topic": https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html#kafka.KafkaConsumer.partitions_for_topic – Alan May 16 '19 at 03:12
As suggested by @Alan try to use : client.cluster.partitions_for_topic(topic) – Soufiane Roui Mar 15 '21 at 15:49

score 3 · Answer 2 · edited Sep 02 '22 at 09:12

3

For those of you using Confluent-Python or the enterprise API. This can be done this way:

cluster_data: ClusterMetadata = producer.list_topics(topic=TOPIC)
topic_data: TopicMetadata = cluster_data.topics[TOPIC]
available_partitions: PartitionMetadata = topic_data.partitions
print("Count of partitions:", len(available_partitions))

edited Sep 02 '22 at 09:12

clemens

16,716
11
50
65

answered Jun 28 '19 at 16:32

Drew Mash

31
2

score 2 · Answer 3 · answered May 07 '15 at 06:34

I found this question while trying to solve this exact same problem. I know the question is old, but here is the solution I came up with (using Kazoo to talk to zookeeper):

from kazoo.client import KazooClient

class KafkaInfo(object):
    def __init__(self, hosts):
        self.zk = KazooClient(hosts)
        self.zk.start()

    def topics(self):
        return self.zk.get_children('/brokers/topics')

    def partitions(self, topic):
        strs = self.zk.get_children('/brokers/topics/%s/partitions' % topic)
        return map(int, strs)

    def consumers(self):
        return self.zk.get_children('/consumers')

    def topics_for_consumer(self, consumer):
        return self.zk.get_children('/consumers/%s/offsets' % consumer)

    def offset(self, topic, consumer, partition):
        (n, _) = self.zk.get('/consumers/%s/offsets/%s/%d' % (consumer, topic, partition))
        return int(n)

score 2 · Answer 4 · answered Aug 04 '21 at 07:03

Python 3.8.10/kafka-python 2.0.2 Solution:

from kafka import KafkaConsumer

def get_partitions_number(server, topic):
    consumer = KafkaConsumer(
        topic,
        bootstrap_servers=server
    )
    partitions = consumer.partitions_for_topic(topic)
    return len(partitions)

partitions_for_topic

Kafka-python get number of partitions for topic

4 Answers4

Linked