0

I am a beginner in learning Kafka and was going through topics and producer. As per my understanding

  1. The topic is just a logical name for a group of partitions and the partitions are spread across the nodes.

  2. Is my understanding correct that for a given topic, lets say there are 5 partitions, then all 5 partitions will be on 5 different brokers. And if there is another topic with 5 partitions, then all the 5 partitions will be on 5 brokers. Effectively for this configuration, each of the 5 brokers would have two partitions with each partition of a topic. Am I right?

  3. Another point while the producer is posting a message and the consumer is consuming, is that, the producer will have a list of brokers configured and will post the message to a topic and the list of brokers. The message will always be written to the leader partition. i.e one of the partition on a broker. The message will then be replicated to all the other partitions on other brokers. In this, case, if the producer is configured with only one broker in the producer configuration, does the message be posted to the leader partition in this case too, even in case the broker configuration is not the same as the leader partition for that topic, ex: topic name - events with 5 partitions on 5 brokers. broker-2 is contains the leader partition but the producer is configured with broker-1 alone.

  4. I also read that the producer can specify the partition name also while posting the message. If this is the case, is it not contradicting that the producer will also post the message to the leader partition and if the producer post the message to a custom partition and if the broker containing the custom partition is down, then the message will not be posted. Also in case of distributed systems, it is not a best practice to nail down a specific partition. Am I missing something here?

  5. Does the consumer also reads from the lead partition or the consumer group assigns different consumers to different partition?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
zilcuanu
  • 3,451
  • 8
  • 52
  • 105
  • Your main understanding is correct. However, I recommend to get familiar again with the difference between a *partition* and *replica* as you are mixing them up in point 3 and onwoards. You could for aexample have a look at [What is the difference between partitions and replicas](https://stackoverflow.com/questions/63167309/what-is-the-difference-between-kafka-partitions-and-kafka-replicas/63167468?%3Fr=SearchResults#63167468). – Michael Heil Jan 30 '21 at 11:06
  • @mike I am still not clear. What is the difference between the producer posting to a specific partition vs the leader partition? – zilcuanu Jan 30 '21 at 11:21
  • 1
    Every client (consumer and producer) only communicates with the partition leader. Every partition has its own leader. – Michael Heil Jan 30 '21 at 11:39
  • I am still having some questions. How shall we communicate and get my doubts clarified? – zilcuanu Jan 30 '21 at 12:01
  • You can edit your question, but you should really limit your posts to single, specific questions. The Kafka documentation should be able to handle most of these answers – OneCricketeer Jan 30 '21 at 13:49

0 Answers0