0

I'm using confluent-kafka KafkaConsumer library for reading data from kafka. Here is the scenario.

  1. Demo topic is compaction enabled.
  2. Consumer-1 is reading data using the latest property with a group.id="group-1".
  3. Now I want Consumer-2 to read all data from the beginning in topic with group.id="group-2".
  4. When I use auto.offset.reset="earliest" it reads data from the beginning, but in case the consumer restarts then it again reads data from the beginning.

How should I configure new consumer-2 so that it will read data from the beginning only once, after this, it will give only the not acknowledged data.

Thanks

Akash Pagar
  • 637
  • 8
  • 22
  • 1
    When a consumer starts and is assigned a partition to consume, it will start at its group’s committed offset or latest or ealiest as auto.offset.reset (When there is a committed offset, the auto.offset.reset property is not used). [Confluent Configuration](https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_auto.offset.reset) – Arun Mar 09 '22 at 07:54
  • is it possible to read data from-beginning with offset commit so that this read data will not come when consumer restarts? – Akash Pagar Mar 09 '22 at 09:24
  • I think i would first check is at what frequency are you commiting offset(if not using the default autocommit flag) and describe a consumer group to check the lag and stuff. Then i would look deep into two more configuration. ` "enable.auto.commit" : "true",// default is true as well "auto.commit.interval.ms" : "1000" ` – teedak8s Apr 17 '22 at 02:23

0 Answers0