18

I have 3 Kafka brokers in 3 different VMs, with one additionally running a Zookeeper. I now create a topic with 8 partitions. The Producer pushes messages to these group of brokers on the created "topic".

  • How does the Kafka distribute a topic and its partitions among the brokers?
  • Does the Kafka redistribute the topic when a new Kafka Broker joins a cluster?
  • Can the topic partition be increased after the topic was created?
vvra
  • 2,832
  • 5
  • 38
  • 82

1 Answers1

32

  • When you create a new topic, Kafka places the partitions and replicas in a way that the brokers with least number of existing partitions are used first, and replicas for same partition are on different brokers.

  • When you add a new broker, it is used for new partitions (since it has lowest number of existing partitions), but there is no automatic balancing of existing partitions to the new broker. You can use the replica-reassignment tool to move partitions and replicas to the new broker.

  • Yes, you can add partitions to an existing topic.

Gwen Shapira
  • 4,978
  • 1
  • 24
  • 23
  • 3
    A precise answer! Thanks for that. Do the producers need to know which broker the partition exists and push message to them? What happens when that broker goes down while the producers are pushing messages? Basically, how the producers see the partitions in a huge Kafka cluster? – vvra Sep 07 '15 at 08:03
  • 6
    When a producer firsts connect to the cluster (via a broker in broker.list parameter), it issues a Metadata request, with the topics you are publishing to, and the broker replies with which broker has which partition. If a broker goes down, the producer detects the failed connection (via tcp timeout usually), and re-issues a metadata request to find the new leader. – Gwen Shapira Sep 07 '15 at 22:53