0

Trying to build a list of possible errors that can potentially happen during the execution of kafkaTemplate.send() method:

  • Errors related serialization process;
  • Some network issues or broker is down;
  • Some technical issues on broker side, for example acknowledgement not received from broker, etc.

And now I need to find a way how to handle all possible errors in the right way:

Based on the business requirements: in case of any exceptions I need to do the following things:

  • Retry 3 times;
  • If all 3 retries failed - log appropriate message.

I found that configuration property spring.kafka.producer.retries available, and I believe it exactly what I need.

But have can I configure recovery method (method that will be executed when all retries failed)?

user471011
  • 7,104
  • 17
  • 69
  • 97

1 Answers1

1

Probably that spring.kafka.producer.retries is not what you are looking for. This auto-configuration property is mapped directly to ConsumerConfig:

map.from(this::getRetries).to(properties.in(ProducerConfig.RETRIES_CONFIG));

and then we go and read docs for that ProducerConfig.RETRIES_CONFIG property:

private static final String RETRIES_DOC = "Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error."
        + " Note that this retry is no different than if the client resent the record upon receiving the error."
        + " Allowing retries without setting <code>" + MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION + "</code> to 1 will potentially change the"
        + " ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second"
        + " succeeds, then the records in the second batch may appear first. Note additionally that produce requests will be"
        + " failed before the number of retries has been exhausted if the timeout configured by"
        + " <code>" + DELIVERY_TIMEOUT_MS_CONFIG + "</code> expires first before successful acknowledgement. Users should generally"
        + " prefer to leave this config unset and instead use <code>" + DELIVERY_TIMEOUT_MS_CONFIG + "</code> to control"
        + " retry behavior.";

As you see spring-retry is fully not involved in the process and all the retries are done directly inside Kafka Client and its KafkaProducer infrastructure.

Although this is not all. Pay attention to the KafkaProducer.send() contract:

Future<RecordMetadata> send(ProducerRecord<K, V> record);

It returns a Future. And if we take a look closer to the implementation, we will see that there is a synchronous part - topic metadata request and serialization, - and enqueuing for the batch for async sending to Kafka broker. The mentioned ProducerConfig.RETRIES_CONFIG has an effect only in that Sender.completeBatch().

I believe that the Future is completed with an error when those internal retries are exhausted. So, you probably should think about using a RetryTemplate manually in the service method around KafkaTemplate to be able to control a retry (and recovery, respectively) around metadata and serialization which are really sync and blocking in the current call. The actual send you also can control in that method with retry, but if you call Future.get() to block it for a response or error from Kafka client on send.

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118