Kafka, Spring Kafka and redelivering old messages

Question

I use Kafka and Spring Boot with Spring Kafka. After abnormal application termination and then restart, my application started receiving the old, already processed messages from Kafka queue.

What may be the reason for that and how to find and resolve the issue?

my Kafka properties:

spring.kafka.bootstrap-servers=${kafka.host}:${kafka.port}
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.group-id=postfenix
spring.kafka.consumer.enable-auto-commit=false

My Spring Kafka factory and listener:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, Post> postKafkaListenerContainerFactory(KafkaProperties kafkaProperties) {

    ConcurrentKafkaListenerContainerFactory<String, Post> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.getContainerProperties().setAckMode(AckMode.MANUAL);
    factory.setConsumerFactory(postConsumerFactory(kafkaProperties));

    return factory;
}

@KafkaListener(topics = "${kafka.topic.post.send}", containerFactory = "postKafkaListenerContainerFactory")
public void sendPost(ConsumerRecord<String, Post> consumerRecord, Acknowledgment ack) {

    Post post = consumerRecord.value();

    // do some logic here

    ack.acknowledge();
}

How do you commit your offsets? – AlexLiesenfeld Jul 08 '18 at 10:21 — AlexLiesenfeld, Jul 08 '18 at 10:21
I have updated the question with my properties – alexanoid Jul 08 '18 at 10:25 — alexanoid, Jul 08 '18 at 10:25

AlexLiesenfeld · Accepted Answer · 2018-07-11T06:53:20.627

9

When using Kafka, the clients need to commit offsets themselves. This is in contrast to other message brokers, such as AMQP brokers, where the broker keeps track of messages a client did already receive.

In your case, you do not commit offsets automatically and therefore Kafka expects you to commit them manually (because of this setting: spring.kafka.consumer.enable-auto-commit=false). If you do not commit offsets manually in your program, the behaviour you describe is pretty much the expected one. Kafka simply does not know what messages your program did process successfully. Each time you restart your program, Kafka will see that your program did not commit any offsets yet and will apply the strategy you provide in spring.kafka.consumer.auto-offset-reset=earliest, which means the first message in the queue.

If this is all new to you, I suggest reading up this documentation on Kafka and this Spring documentation, because Kafka is quite different than other message brokers.

edited Jul 11 '18 at 06:53

answered Jul 08 '18 at 10:31

AlexLiesenfeld

2,872
7
36
57

Thank you very much for such a detailed answer. I have updated my Spring Kafka factory and listener. Added `AckMode.MANUAL` to the factory and `Acknowledgment ack` to the listener. Could you please check that I did it correctly in order to avoid the main error described in my question? Thanks! – alexanoid Jul 08 '18 at 11:02
1

These are all Spring constructs which Spring adds on top of what Kafka provides, but yes, they should be exactly what you need to commit manually. Please compare to the following post go get more information on how to use the Acknowledgement object correctly: https://stackoverflow.com/questions/46325540/how-to-use-spring-kafkas-acknowledgement-acknowledge-method-for-manual-commit – AlexLiesenfeld Jul 08 '18 at 11:12
2

If you set the `AckMode.RECORD`, the container will commit the offset for you when the listener returns normally; the default AckMode is BATCH, which means the offsets will be committed when all the results of the poll have been processed. This reduces network activity but can mean duplicate deliveries are possible after a failure. `AckMode.MANUAL` similarly does the commits at the end of the batch. With recent versions of spring-kafka (>= 1.3), `AckMode.MANUAL_IMMEDIATE` means that `acknowledge()` will cause the offset to be committed immediately. – Gary Russell Jul 08 '18 at 13:49

Kafka, Spring Kafka and redelivering old messages

1 Answers1