I am writing a script to reset the offsets of a reused consumer group to the start of a topic. This is the command I am trying to simulate from the client side using python and confluent_kafka:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group your-group-name --reset-offsets --to-earliest --execute --topic your-topic-name
I have tried a number of things, but either the offsets do not get committed, or only a single partition gets reset.
Example of what I have tried:
consumer.subscribe([topic])
msg = None
while msg is None:
msg = consumer.poll(10)
partitions = consumer.assignment()
for partition in partitions:
partition.offset = confluent_kafka.OFFSET_BEGINNING
consumer.commit(offsets=partitions, asynchronous=False)
# KafkaError: Commit Failed: No offset stored
I have also tried doing a custom on_assign that sets assigns the TopicPartition with an offset of OFFSET_BEGINNING, but that requires consuming messages and the offset will not be the start by the time all partitions are set up.
