How can I commit a message offset to Kafka topic after consumption?

Question

It seems that the offset is not being committed.

I am using kafka Python package. This is my code

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'quickstart-events',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',
    enable_auto_commit=True,
    group_id='my-group',
    auto_commit_interval_ms=1000,
)

msg_pack = consumer.poll(max_records=10,timeout_ms=500,update_offsets=True)
for tp,messages in msg_pack.items():
    for message in messages:
        print("%s:%d:%d: key=%s value=%s" % (tp.topic, tp.partition,
                                             message.offset, message.key,
                                             message.value))

But the problem is I always get all the messages from the starting. Sometimes I don't get any. Can you please help me?

quickstart-events:0:52: key=None value=b'1'
quickstart-events:0:53: key=None value=b'2'
quickstart-events:0:54: key=None value=b'3'
quickstart-events:0:55: key=None value=b'4'
quickstart-events:0:56: key=None value=b'5'
quickstart-events:0:57: key=None value=b'1'
quickstart-events:0:58: key=None value=b'2'
quickstart-events:0:59: key=None value=b'3'
quickstart-events:0:60: key=None value=b'4'
quickstart-events:0:61: key=None value=b'5'

I would reword your question to a Kafka only question as it involves no aspect of AWS or unique aspect of SQS — Ermiya Eskandary, Oct 25 '21 at 16:37
Kafka and SQS are not really equivalent. It's hard to tell what you expect — OneCricketeer, Oct 25 '21 at 16:41

score 1 · Accepted Answer · answered Oct 25 '21 at 16:39

1

I always get all the messages from the starting.

This should only happen once, based on these

auto_offset_reset='earliest',
enable_auto_commit=True,
group_id='my-group',

Sometimes I don't get any

Then you might have no lag in the consumer group (no records to consume). You'd need to use kafka-consumer-groups --describe --group my-group to check the lag or that offsets are getting committed in the way you expect.

If offsets aren't committed in the way you expect, then you should disable auto commits and do it manually

answered Oct 25 '21 at 16:39

OneCricketeer

179,855
19
132
245

thanks for your answer, but it does not work. Can you please edit how we can update the offset after each message is processed? – jebaseelan ravi Oct 25 '21 at 16:44
1

What doesn't work? `kafka-consumer-groups` CLI? Have you tried calling [`consumer.commit`](https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html#kafka.KafkaConsumer.commit) within the loop? – OneCricketeer Oct 25 '21 at 16:58
1

Yeah , i tried adding the `consumer.commit()` inside the loop. It works Thanks – jebaseelan ravi Oct 25 '21 at 17:05

How can I commit a message offset to Kafka topic after consumption?

1 Answers1