-1

I have 5 broker kafka version 0.10 cluster.Replication factor is 3. and this is production kafka

brokers IDS are

101
102
103
104
105

after couple months that cluster was ok , we observed following logs in Kakfa server.log.

from the log we can see many lines of 'This server is not the leader for that topic-partitionb' exception.

the topic - kopa.thrn.bvff have 100 partitions

and we can see that all 100 partitions are balanced and no need to run kafka kafka-reassign-partitions

What may be the possible reason?

Please help me.

[2023-01-19 11:53:37,434] ERROR [ReplicaFetcherThread-0-101], Error for partition [kopa.thrn.bvff,78] to broker 101:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2023-01-19 11:53:37,434] ERROR [ReplicaFetcherThread-0-101], Error for partition [kopa.thrn.bvff,23] to broker 101:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2023-01-19 11:53:37,434] ERROR [ReplicaFetcherThread-0-101], Error for partition [kopa.thrn.bvff,63] to broker 101:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2023-01-19 11:53:37,434] ERROR [ReplicaFetcherThread-0-101], Error for partition [kopa.thrn.bvff,98] to broker 101:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2023-01-19 11:53:37,434] ERROR [ReplicaFetcherThread-0-101], Error for partition [kopa.thrn.bvff,3] to broker 101:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
Judy
  • 1,595
  • 6
  • 19
  • 41

1 Answers1

3

The leader broker and the follower brokers manage each partition in Kafka. Since you have replication factor 3, each partition will have one leader broker and 2 follower brokers.

When the Kafka producer produces data, it connects to the leader and puts the data there, the followers will copy the data from the leader.

Now, the Kafka leader broker can be reassigned based on the leader's availability, if the leader was unavailable for some time for any reason in a distributed environment (busy CPU, network partition etc), Kafka will run the leader election for the partition to elect a leader for the partition.

You can see who is the leader and who is the follower by topic describe command.

In your case, the partition leader has been changed due to some unavailability of the leader. If you have Kafka metrics, you could see those leader election events for the partition. It is hard in a distributed environment to ensure one broker will remain the leader forever.

Avishek Bhattacharya
  • 6,534
  • 3
  • 34
  • 53
  • first thank you so much for this long explain , regarding to what you said " unavailability of the leader" , can we get the reason for why leader became unavailability ? – Judy Jan 23 '23 at 08:02
  • 1
    Hi @Judy as Avishek mentioned, the leader can become unavailable for a number of reasons - network problems, busy periods may cause a heavy load that the broker can't handle etc. – Jessica Vasey Jan 23 '23 at 13:11
  • @ Jessica Vasey can you explain the meaning of "busy periods" – Judy Jan 24 '23 at 05:59
  • 1
    When the broker receives multiple requests from the producers, the broker needs to process those requests. So the CPU utilization of the broker increases and thus it can stop/delay sending heartbeat messages to the other follower brokers. This can make the follower broker to realize that the leader broker has died and thus create a new leader election process. – Avishek Bhattacharya Jan 24 '23 at 06:30
  • regarding to "broker receives multiple requests from the producers," , what is the indication for that ? ., or any sign from kafka cluster that this happened beside the CPU idle – Judy Jan 24 '23 at 09:01
  • 1
    You can see kafka metrics what is the producer load, what are the CPU memory of the brokers, if the leader is network partitioned etc. To monitor you may use : https://grafana.com/solutions/kafka/monitor/?src=ggl-s&mdm=cpc&camp=nb-kafka-exact&cnt=142948784315&trm=grafana%20kafka%20consumer%20dashboard&device=c&gclid=EAIaIQobChMIiOG30v7f_AIVjH8rCh049QhfEAAYASAAEgLLA_D_BwE – Avishek Bhattacharya Jan 24 '23 at 10:24