1

In Kafka, is it possible to set a backoff time per message, if processing of that message fails ? - So that I can process other messages, and try again later with the message that failed ? If I just put it back out in front of the topic, it will reappear very fast. I am using Kafka with Spring Boot.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
user1511956
  • 784
  • 3
  • 9
  • 22

1 Answers1

1

Kafka does not have any built-in capabilities for a backoff time when consuming data as far as I know.

As soon as you process other messages successfully and also commit them it will be difficult to re-read only those where the processing failed. Kafka topics are built to be consumed in sequence while guaranteeing the order of messages per TopicPartition.

What we usually do in such a scenario is to catch the Exception during the processing of the message and then send it into a seperate topic (together with a error code/hint) and continue the processing of later incoming messages. That way you can analyse the data later and, if necessary, move the messages from that other topic into your original topic again.

The insertion of the problematic messages from the seperate topic into your original input topic could be done through a simple batch job that you run from time to time or even using the command line tools provided by Kafka.

Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • So I basically need a producer with functionality that pushes a failed message on the dead letter topic, and then I need a callback to make sure that the message was pushed onto the DL topic ? And I guess that I need functionality for periodically transferring messages from the DL topic to the main topic, and error handling for this as well ? – user1511956 Sep 18 '20 at 08:01
  • First question: You application now needs to produce messages to two different topics (dead letter and regular topic). HOW you make sure that the data was sucessfully produced is another question. You may want to look into the producer configurations `acks` and also see if you rather want synchronous or asynchronous modues (see [here](https://stackoverflow.com/questions/63483418/kafka-producer-losing-message-even-if-set-acks-all/63483711#63483711) for more details). – Michael Heil Sep 18 '20 at 08:11
  • Second question: Yes, you need some functionality to periodically transfer messages from DL topic to main topic. I have adjusted my answer to give you options (at least that is how we do it). And yes, I would also recommend to have proper error handling for that transfer in place. – Michael Heil Sep 18 '20 at 08:13
  • Ok, I will look into this. Thank you for very good answers – user1511956 Sep 18 '20 at 08:16