3

I have a spring boot application, that keeps reading all messages from a specific topic. My problem is that I have an issue that when the application restarts, it needs to read all the messages from the topic again. I thought these two options combined would do it, but it is not working:

  • resetOffsets: true
  • startOffset: earliest

Here is my method;

 @StreamListener(myTopic)
 public void handle(@Payload Input input) {
    /** Do other stuff not related **/
 }

Here is my application.yaml

spring:
  cloud.stream:
    bindings:
      myTopic:
        destination: TOPIC
        content-type: application/**avro
        group: group-topic
        concurrency: 1
        resetOffsets: true
        startOffset: earliest
Türkmen Mustafa Demirci
  • 333
  • 5
  • 13
Victor
  • 47
  • 8
  • You need to change the consumer group (configuration `group`) to be able to read from beginning. Otherwise the two options resetOffset and startingOffset will not have any impact. – Michael Heil Nov 09 '20 at 23:33
  • I couldnt understand what I have to change. Could you detail a little more? Sorry, I'm new at Kafka. – Victor Nov 10 '20 at 14:16
  • Hi @Victor no problem. Have added more details in the answer below. – Michael Heil Nov 10 '20 at 14:20

1 Answers1

2

You need to change the group within your application.yaml to a new and unique group name (see below example where I set the consumer group to new-consumer-group-id:

spring:
  cloud.stream:
    bindings:
      myTopic:
        destination: TOPIC
        content-type: application/**avro
        group: new-consumer-group-id
        concurrency: 1
        resetOffsets: true
        startOffset: earliest

If you keep on using the same ConsumerGroup, the configurations resetOffset and startingOffset will not have any impact.

As an alternative, you could reset the offsets for each consumer group using the command line tool kafka-consumer-groups.sh. A template is shown below:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
 --execute --reset-offsets \
 --group groupA\
 --topic topicC \
 --partition 0 \
 --to-offset <insert number>
Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • So in this case, every time my application starts it needs to have a different consumer group? Like, adding a timestamp at the end of the name defined in the yaml? – Victor Nov 10 '20 at 14:26
  • yes, that would be a solution. Otherwise you can make use of a command line tool that comes with Kafka. I have updated my answer accordingly. – Michael Heil Nov 10 '20 at 14:28
  • Hi @mike, yes, this answered. Thanks a lot. – Victor Nov 16 '20 at 20:19