1

I have found this question that talks about the difference between partition and replica, and the answers seem to mention that the Kafka partition is needed for scalability. But I don't get why it's "mandatory" in order to scale your infrastructure? I feel like you could simply add a new node and increase the replication value of the topic?

Simone
  • 20,302
  • 14
  • 79
  • 103
Kevin Heirich
  • 109
  • 2
  • 12
  • 2
    AFAIK a single partition is always stored on a single node. So at the very least you can't scale the size of a topic to more than one node can handle unless you split it into more than one partition. There are probably other constraints as well, but that's one basic one that's easy to understand. – Generous Badger May 25 '21 at 09:29

1 Answers1

3

Consumer Application side Scalability

Partitions are not shared within same group consumer instances. If your topic has only one partition, And your consumer application has multiple instances with same consumer group id it is useless. So if you need to scale your consumer application to multiple instances, you need to have multiple partitions.

Kafka Broker side Scalability

And if your topic is too busy with messages, if you have multiple partition, you can add another node and rebalance partitions so they will be shared with new brokers. So, broker traffic will be shared with multiple partitions. If you have only one partition, no traffic is shared, making that not scalable.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
nipuna
  • 3,697
  • 11
  • 24