pykafka consumes same messages

Question

Pykafka Kafkaconsumer consumes same messages from beginning everytime.

def consume_from_kafka():
    client = KafkaClient(hosts=IP)
    topic = client.topics[TOPIC]
    consumer = topic.get_simple_consumer(consumer_group="mygroup",reset_offset_on_start=True)
    consumer.commit_offsets()
    for message in consumer:
        if message is not None:
            print(message.offset, message.value)

1) `PyKafka` is no longer maintained. You may want to test with other libraries such as `kafka-python` or `confluent-kafka-python` 2) What do you think `reset_offset_on_start=True` does? — OneCricketeer, Mar 31 '22 at 21:44

score 0 · Answer 1 · edited Apr 01 '22 at 13:20

reset_offset_on_start=True

https://pykafka.readthedocs.io/en/latest/usage.html

For any existing group/topic/partitions, assuming reset_offset_on_start=False, consumption will start from the offset immediately following the last committed offset (if the last committed offset was 4, consumption starts at 5). If reset_offset_on_start=True, consumption starts from auto_offset_reset. If there is no committed offset, the group/topic/partition is considered new.

Move consumer.commit_offsets() after the loop, or into it

pykafka consumes same messages

1 Answers1