-2

I'm using sarama-cluster (written by Golang kafka consumer client)

In broker, my topic's partition offset was 11000 and my consumer group's partition offset was 10100. Then I run my cluster-consumer, but nothing consume. (consume time was 1~2days later)

But when I produce message in the topic's partition, it consume! (In each partition)

A number of message is 901. Why is it, that my consumer-cluster consume seems to activate when produce message?

My consumer setting was auto.offset.reset = lastest

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
David Kang
  • 1
  • 2

1 Answers1

0

This is because of your offset reset settings. auto.offset.reset = latest means your consumer group should wait for the newest records. If you want to consume from the beginning, use auto.offset.reset = earliest.

The official Kafka documentation: https://kafka.apache.org/0110/documentation.html

codejitsu
  • 3,162
  • 2
  • 24
  • 38
  • But it means Consumer will start consuming at offset no 0 ???? right? I wanna consume offset no.10101 (in my example case) – David Kang Mar 05 '18 at 05:24
  • should I commit offset number? – David Kang Mar 05 '18 at 08:55
  • I'm afraid it would be difficult to read from specific offset, I guess. Usually your consumer reads the data from the last _known_ offset. If offset is unknown, then the _auto.offset.reset_ policy kicks in. – codejitsu Mar 05 '18 at 12:01
  • maybe I need this document. https://kafka.apache.org/0110/documentation.html#impl_offsettracking – David Kang Mar 06 '18 at 18:54