3

Recently, I began to study working with kafka. The project I'm working on uses sarama.

For reading messages I use ConsumerGroup.

I need to read the message again after some time if foo returns false. How can this be done?

func (consumer *Consumer) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {

    for message := range claim.Messages() {

            if ok := foo(message); ok {
                session.MarkMessage(message, "")
            } else {
                // ???
            }

    }

    return nil
}
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
rogatzkij
  • 33
  • 1
  • 6
  • You cannot. That is not how Kafka works. If you do not commit any more messages and restart you will be able to consume from the last commited offset. Probably the simplest thing is to to publish the message once more. – Volker May 04 '20 at 13:35
  • If I do not want to publish new messages, then to read the message again, will I have to create a new `ConsumeGroup` each time with a set offset? – rogatzkij May 04 '20 at 14:27

1 Answers1

5

You can reset the offset of a Consumer Group to an older offset by including the following in your Consumer Group's Setup() callback:

func (e myConsumerGroup) Setup(sess sarama.ConsumerGroupSession) error {
    sess.ResetOffset(topic, partition, offset, "")

    return nil
}

You can also achieve the same through console: 

kafka-consumer-groups \
    --bootstrap-server localhost:9092 \
    --group my-consumer-group \
    --topic myTopicName \
    --reset-offsets \
    --to-offfset 100 \
    --execute
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156