12

My Apache Kafka producer (0.9.0.1) intermittently throws a

org.apache.kafka.common.errors.NotLeaderForPartitionException

My code that performs the Kafka send resembles this

final Future<RecordMetadata> futureRecordMetadata = KAFKA_PRODUCER.send(new ProducerRecord<String, String>(kafkaTopic, UUID.randomUUID().toString(), jsonMessage));

try {
    futureRecordMetadata.get();
} catch (final InterruptedException interruptedException) {
    interruptedException.printStackTrace();
    throw new RuntimeException("sendKafkaMessage(): Failed due to InterruptedException(): " + sourceTableName + " " + interruptedException.getMessage());
} catch (final ExecutionException executionException) {
    executionException.printStackTrace();
    throw new RuntimeException("sendKafkaMessage(): Failed due to ExecutionException(): " + sourceTableName + " " + executionException.getMessage());
}

I catch NotLeaderForPartitionException within the catch (final ExecutionException executionException) {} block.

Is it OK to ignore this particular exception?

Has my Kafka message been sent successfully?

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Hector
  • 4,016
  • 21
  • 112
  • 211

1 Answers1

25

If you receive NotLeaderForPartitionException, your data was not written successfully.

Each topic partition is stored by one or multiple Brokers (with one leader; the remaining brokers are called followers) depending on your replication factor. A producer needs to send new messages to the leader Broker (data replication to followers happens internally).

Your producer client does not connect to the correct Broker, ie, to a follower instead of the leader (or to a broker that is not even a follower any longer), and this broker rejects your send request. This can happen if the leader changed but the producer still has outdated cached metadata about which broker is the leader for a partition.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • 1
    What is the solution for it? – Vinit Gaikwad Aug 06 '18 at 18:58
  • 5
    @VinitGaikwad The producer will retry sending the data internally first (and will refresh it's metadata to learn about the new leader the data must be sent to) -- thus, the application get this exception only if all retries are exhausted. Hence, you might want to increase the `retries` config parameter for the producer. If you want to handle it at the application level (was seems to be the second best option compared to increasing producer retries), you would need to call `Producer.send()` to resend the data. – Matthias J. Sax Aug 06 '18 at 20:50