8

Kafka consumer has processed the messages 1, 2, 3, 4 and the enable.auto.commit is set to false.

But on restarting the consumer, it is not reprocessing the above messages again, from CLI I could see that the offset has been incremented and there is no lag (hence it is committing).

Can you please help on this, to understand how the consumer is still committing the offsets though the property enable.auto.commit is set to false.

Below are the consumer properties

allow.auto.create.topics = true
auto.commit.interval.ms = 0
auto.offset.reset = latest
bootstrap.servers = [localhost:9092]
enable.auto.commit = false
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = EmployeeConsumer

Currently using the spring-kafka-2.5.0.RELEASE.jar as the dependency

Harsha
  • 349
  • 1
  • 8
  • 20

1 Answers1

20

You need to show your Spring configuration.

enable.auto.commit=false tells the kafka-clients not to commit offsets, but Spring will commit offsets by default.

Set the listener container ackMode property to AckMode.MANUAL to disable the container commits.

Furthermore:

auto.offset.reset = latest means that a consumer that has never committed an offset will start consuming from the current end of the topic/partition so it won't get existing records.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • @Bean public ConcurrentKafkaListenerContainerFactory kafkaListenerContainerFactory() { ConcurrentKafkaListenerContainerFactory factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.getContainerProperties().setAckMode(AckMode.MANUAL); factory.setConsumerFactory(consumerFactory()); return factory; } It worked! – Harsha May 20 '20 at 04:33
  • @Gary Russell So if both are set to true then do they override each other at time of storing offsets? – Ash3060 May 17 '21 at 15:29
  • 1
    I am not sure what you mean. Only one of the described properties is boolean. If auto.offset.reset is true, Spring is hands off since you want the kafka-clients to be in control. This is not recommended. Spring uses a more determinative algorithm. – Gary Russell May 18 '21 at 00:27