4

Let's assume I have a Kafka Cluster of 3 brokers and 3 zookeeper.

I have a single topic accountsTopic which is setup as replication factor of 3 and ISR of 2.

If the Leader dies, kafka will elect the ISR as leader, but the 3rd one, (which say for arguments sake was not ISR), is it going to serve as a ISR of current leader in this 2 node fail-over setup?

Divs
  • 1,578
  • 2
  • 24
  • 51

1 Answers1

3

What happens when the leader dies depends on your configuration.

By default since 0.11, only one of the replicas in-sync can be elected as a leader. If no replicas are in-sync, the partition goes offline. This is favoring consistency over availability.

You can set unclean.leader.election.enable=true on your brokers and in this case, if no replicas are in-sync, one of the out-of-sync replicas will be elected. This can lead to data loss but is favoring availability. Of course if some replicas are in-sync it will still elect one of them.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • In case it is not clear in the question, my `acks` setting will always ensure the ISR replica is written into before commiting the message as written. So, in short, let's assume in my case, I will always have a `ISR` replica for a clean leader election. However, I want to understand what happens to the `Non-ISR` broker after the new leader election? Does it catch up and become a `ISR` of the new leader? – Divs Jul 05 '18 at 04:39
  • 1
    An out-of-sync replica is always trying to get back in-sync by fetching data from the leader. If the leader changes, it will try to catch up from the new leader – Mickael Maison Jul 05 '18 at 08:45
  • Thanks. It will certainly try to catch up, but when a `write` happens to the new leader, is it guaranteed that the 3rd one will become new `ISR` of the new leader, so that a write to the new leader will be considered committed _only_ if the write in 3rd one also succeeds? – Divs Jul 05 '18 at 11:15
  • I think you're confused about what ISR means. ISR = In-Sync Replica. When a broker has replicated all messages from the leader it is in-sync. So yes once the 3rd broker catches up with the new leader it will become an ISR. Regarding writes, it depends on your settings. With acks=all, all the ISR brokers need to have replicated the message to be considered a successful send. But depending on `min.insync.replicas` and the replication factor of the topic it can require a different number of brokers. See the [broker configuration docs](http://kafka.apache.org/documentation/#brokerconfigs) – Mickael Maison Jul 07 '18 at 10:25
  • @MickaelMaison In my test with 3 Broker, Replication factor 2, min ISR 2, ack 1. broker 0 goes down, partition remains with 1 ISR so write is not be permitted with NotEnoughReplicasException. Will another remaining broker become ISR? after how much time? here you mentioned that [Kafka does not create a new replica when a broker goes down](https://stackoverflow.com/questions/47132158/what-happens-after-a-broker-is-down-in-a-cluster). – Girish Dec 08 '20 at 12:12