How to identify which messages successfully published to kafka topic and those which failed

Question

Publishing a list of messages to apache kafka. Could anyone provide sample code using the kafka api showing how to identify which messages got successfully published to the topic and those which failed from the response? (Note I'm sending the list of messages as a batch in one request.)

Answer 1

The KafkaProducer.send() method takes a single ProducerRecord (message).

There are 2 ways to check if this message is successfully received by the cluster:

• Use a Callback: send() can take a Callback as the 2nd argument

  ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
  producer.send(record, new Callback() {
      @Override
      public void onCompletion(RecordMetadata metadata, Exception exception) {
          // If Exception is null, the record was sent successfully
      }
  });
  

• Use the Future: send() returns a Future

   ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
   Future<RecordMetadata> future = producer.send(record);
   try {
       RecordMetadata rm = future.get();
       // The record was sent successfully
   } catch (ExecutionException e) {
       // The record failed
   }