Ideal number of Kafka topics per Kafka cluster

Question

We have a use case where we would like to have around 2000 kafka topics (we have 3 brokers) and 1 partition per topic. Do we have any known recommendation on what is the maximum number of topics that one cluster can have? I went through the following url but it is about partitions but not number of topics.

https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/

Would there be any additional overhead on management of the cluster if we have more topics?

I have done a lots of searches on web but did not find any recommendations or known limitations w.r.t. high number of kafka topics.

score 1 · Answer 1 · answered Mar 01 '23 at 15:09

1

2000 is fine. Yes, there will be more and more overhead for each partition (not only topic) that's managed, as partitions are replicated and written to disk, causing network and disk IO for each and every one of them.

1 partition for every topic in your cluster is not recommended. (the consumer offsets topic has more than that, anyway)

Can I have 100s of thousands of topics in a Kafka Cluster?

answered Mar 01 '23 at 15:09

OneCricketeer

179,855
19
132
245

Hi , do you know the impact of the partition numbers to the producer? – rosa Mar 02 '23 at 08:49
Very little, because the producer will only send to a few partitions in any given request – OneCricketeer Mar 02 '23 at 17:36

score -1 · Answer 2 · answered Mar 01 '23 at 06:57

-1

for MSK there is limit for partition per broker.

answered Mar 01 '23 at 06:57

rosa

87
1
3

Question wasn't asking about MSK – OneCricketeer Mar 01 '23 at 15:06

Ideal number of Kafka topics per Kafka cluster

2 Answers2