3

I have an issue with kafka producer in production where I see below error.

"Publish failed, Expiring 2 record(s) for myTopic-1:120004 ms has passed since batch creation[[ org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for myTopic-1:120004 ms has passed since batch creation"

Kafka brokers are of confluent 5.3.2 version and the kafka-client is apache 2.3.1. Producer config which are explicitly specified in my code are below and remaining are defaults.

batch.size = 102400 linger.ms = 100 compression.type = lz4 ack = all

Sample Java Code

ProducerRecord<String, String> rec = new ProducerRecord<String, String>("myTopic",1,"myKey","json-payload-here");
producer.send(rec, new ProducerCallback(jsonPayload));  

private class ProducerCallback implements Callback {

  private String _ME ="onCompletion";
  private String jsonPayload;

  public ProducerCallback(String jsonPayload) {
    this.jsonPayload = jsonPayload;
  }

  @Override
  public void onCompletion(RecordMetadata recordMetadata, Exception e) {
    if (e == null) {
      LOG.logp(Level.FINEST, _CL, _ME, "Published kafka event "+jsonPayload);
    } else {
      //Note: Exception is logged here.
      LOG.log(Level.SEVERE, "Publish failed, "+e.getMessage(), e);
    }
  }
}

Couple of questions

  1. Load is not heavy in production and its moderate as of now and might be heavy in later stages. Am I missing some producer config to rectify above issue?
  2. Assuming 2 records has been expired in the batch, Is there a way I can get those expired records in java so that I can get payload and key to republish them?

Thanks, appreciate your help in advance.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Robin Kuttaiah
  • 91
  • 1
  • 5

1 Answers1

3

The batch expires, so no, cannot get the data back unless you saved the data in some other data structure.

To actually send the batch, can lower the batch size or your can explicitly call producer.flush(). To increase the duration of the timeout, use request.timeout.ms.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • How do I know which record in the batch is expired so that I can fetch it from saved location? – Robin Kuttaiah Jun 03 '22 at 16:50
  • Is it like the entire batch is expired or only 2 records(in my case) in the batch got expired and the rest has been pushed to broker? – Robin Kuttaiah Jun 03 '22 at 16:52
  • As the error says - "since batch creation". So, yes, it is possible that some records from a previous batch were sent, while the newer batch was not, and expired since producer was not flushed. I don't think there is a way to know exactly which records were lost. – OneCricketeer Jun 03 '22 at 17:25