Should topic partitions be replicated across all broker nodes in a Kafka cluster?

Question

Though answers are available to question similar to above. My curiosity lies in the fact that suppose n1-5 nodes are in cluster where topics t1 is on n1,n2 and n3, topic t2 is on n3,n4,n5. Now if suppose p1 pushes messages in t1 and c1 consumes from t1 and similarly p2 and c2 for t2.

Here is where I have certain doubts?

1. Assume nodes n3- n5 are all down, now still p1 and c1 will have active connection to cluster which is kind of useless as anyways publishing and consuming fails. (metric connection_count is greater than 0 means there are connections to cluster from either producer or consumer)

2. Is it correct way to replicate a topic to all nodes in a Kafka cluster?

3. Why do we give multiple node address details in bootstrap server property is one address is sufficient?

Note: I am a beginner in Kafka world and still experimenting with local setup to discover potential problems which might occur in real world.

Answer 1

1. Why should it fail? Nodes n1 and n2 are still up and running and assuming that the topic had a replication-factor=3 all the data should still be accessible.

2. I'd say it depends. It won't harm to replicate the topics across all nodes but sometimes it is redundant (especially when you have a very high number of brokers in the cluster). For high availability, you should set at least replication-factor=3. This allows for example one broker to be taken down for maintenance and one more to fail unexpectedly.

3. bootstrap.servers is used to setup the connection the Kafka cluster. One address is typically enough to access the whole cluster, but it is always best to provide all the addresses in case one of the servers is down. Note that clients (producers or consumers) make use of all brokers irrespective of which servers are specified in bootstrap.servers.

---

Example of 2 topics (each having 3 and 2 partitions respectively):

Broker 1:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|     Topic 2       |
|   Partition 1     |
+-------------------+

Broker 2:

+-------------------+
|      Topic 1      |
|    Partition 2    |
|                   |
|                   |
|     Topic 2       |
|   Partition 0     |
+-------------------+

Broker 3:

+-------------------+
|      Topic 1      |
|    Partition 1    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

Note that data is distributed (and Broker 3 doesn't hold any data of topic 2).

Topics, should have a replication-factor > 1 (usually 2 or 3) so that when a broker is down, another one can serve the data of a topic. For instance, assume that we have a topic with 2 partitions with a replication-factor set to 2 as shown below:

Broker 1:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

Broker 2:

+-------------------+
|      Topic 1      |
|    Partition 0    |
|                   |
|                   |
|     Topic 1       |
|   Partition 0     |
+-------------------+

Broker 3:

+-------------------+
|      Topic 1      |
|    Partition 1    |
|                   |
|                   |
|                   |
|                   |
+-------------------+

Now assume that Broker 2 has failed. Broker 1 and 3 can still serve the data for topic 1. So a replication-factor of 3 is always a good idea since it allows for one broker to be taken down for maintenance purposes and also for another one to be taken down unexpectedly. Therefore, Apache-Kafka offers strong durability and fault tolerance guarantees.

Note about Leaders: At any time, only one broker can be a leader of a partition and only that leader can receive and serve data for that partition. The remaining brokers will just synchronize the data (in-sync replicas). Also note that when the replication-factor is set to 1, the leader cannot be moved elsewhere when a broker fails. In general, when all replicas of a partition fail or go offline, the leader will automatically be set to -1.

Answer 2

Assume nodes n3- n5 are all down, now still p1 and c1 will have active connection to cluster which is kind of useless as anyways publishing and consuming fails. (metric connection_count is greater than 0 means there are connections to cluster from either producer or consumer)

Answer: If all of the three brokers that is your topic replicas are down, then you cannot produce or consume from that topic. To avoid this kind of situations it is recommended to locate brokers in different racks and provide broker.rack information in broker configs.

broker.rack: Rack of the broker. This will be used in rack aware replication assignment for fault tolerance. Examples: RACK1, us-east-1d

Is it correct way to replicate a topic to all nodes in a Kafka cluster?

Answer: It is totally up to your fault tolerance needs. If you replicate topic to all 6 brokers, then you can tolerate up to 5 broker failures. (of course min.insync.replicas and acks configs are also important. If number of replicas is 6, min.insync.replicas=2, acks=all then you can tolerate up to 4 broker failures to continue sending messages)

Why do we give multiple node address details in bootstrap server property is one address is sufficient?

Answer:bootstrap.servers config is used to initial connection to the Kafka cluster. Yes, one address is enough, but what if the broker in this address is down. You cannot connect to cluster. So it's recommended to provide more than one address, to avoid this kind of situation with redundancy.