1

We are using Spring boot kafka listener the prblem is whenever the pod stops or restarts it looses the offset and then stop processing the messages where it left, this happens with a single pod(if it goes down for some reason) as well as all the pods when restarted. The configuration we are using for the consumer are:

consumerProperties:
        max.poll.records: 300
        auto.offset.reset: earliest
        session.timeout.ms: 300000
        request.timeout.ms: 400000
        allow.auto.create.topics: false
        heartbeat.interval.ms: 80000



bindings:
    input:
      destination: some-input
      group: somegroup
      consumer:
        maxAttempts: 2
        autoCommitOffset: false

We are manually acknowledging using header. Auto commit is set to false When the pods are restarted we see a spike on grafana but it does not processes any of the pending messages of the first run, however when i push new messages those start to get processed, which means that the offset is lost between restarts, is there a missing property. Any help or suggestion is greatly welcomed.

EDIT: It seems like the acknowledgement header is not received in the consumers.. not sure why though

-Vaibhav

vaibhav
  • 3,929
  • 8
  • 45
  • 81
  • Hey , u should use tolatest instead of earliest as offset setting. – SauriBabu Jul 05 '20 at 14:42
  • earliest starts from the first unconsumed message so i don't think that should be a problem here – vaibhav Jul 05 '20 at 16:42
  • No, earliest starts from the start of the topic. However the auto.offset.reset configuration is ONLY used when a consumer starts and there is no committed offset for its group for that partition. – Chris Jul 05 '20 at 18:04
  • Just to be clear, while a consumer is alive it uses an in-memory offset. After a rebalance a consumer starts from the last committed offset if there is one, otherwise the last resort is to refer to auto.offset.reset configuration – Chris Jul 05 '20 at 18:06
  • which means in either of the scenarios the pending messages after the restarts should have been accounted for, i am using spring cloud config store, could there be an issue with group assignment to a consumer after the pod restarts? assuming that while the new pod is spinning sometimes the older pod takes a while to get killed. – vaibhav Jul 06 '20 at 01:58

2 Answers2

3

First, a couple of things to take note:

autoCommitOffset: false

autoCommitOffset: false disables container offset commits - it means you must use the header to commit offset(s). The name of this property is unfortunate because it is similar to the kafka property. There is an open issue to rename it. Set it to true for the container to do the commits.(Updated as per Gary's comments(Thanks Gary))

Setting autoCommitOffset to true, means you are letting the listener container manage the commits for you, which is the recommended practice. See autoCommitOffset and related committing offsets.

auto.offset.reset: earliest

This property only kicks in if the consumer group does not have any valid offset committed in Kafka. Whenever there is an offset committed, the auto.offset.reset is ignored.

Now, as @Chris has mentioned, there are 2 type of offsets, 1 that is kept and managed in-memory by Kafka and that determines what messages are being retrieved on the next poll(), and another one that is committed, in your case, by you, when you manual acknowledge. As you have probably guessed, the in-memory offset is gone when there is a stop/restart. So, the key is to know where is the committed offset pointing to, after the restart.

The AckMode that you are using probably holds the answer for what you are experiencing. Also, before and after the pod restart, can you query the group, to see if there are any lags shown on your topic? In Windows, the command would be something like:

kafka-consumer-groups.bat --bootstrap-server localhost:9092 --describe --group your_group_name
jumping_monkey
  • 5,941
  • 2
  • 43
  • 58
  • 1
    `autoCommitOffset: false` disables container offset commits - it means you must use the header to commit offset(s). The name of this property is unfortunate because it is similar to the kafka property. There is an [open issue to rename it](https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/877). Set it to true for the container to do the commits. – Gary Russell Jul 06 '20 at 15:24
0

Expected configuration was spring.cloud.stream.kafka.default in spite of this we were spring.cloud.stream.default blame it on having too many nested properties in yaml :( this worked

vaibhav
  • 3,929
  • 8
  • 45
  • 81