2

I have a way of performing this task in shell: How to make kafka consumer to read from last consumed offset but not from beginning

However, I am willing to do this in Python, using kafka-python

I could not find any api for this case.

http://kafka-python.readthedocs.io/en/latest/apidoc/KafkaConsumer.html

hasherBaba
  • 319
  • 5
  • 18

1 Answers1

2

To enable Consumer Groups with kafka-python you just need to set the group_id configuration in the Consumer.

From the Consumer API:

group_id (str or None) – The name of the consumer group to join for dynamic partition assignment (if enabled), and to use for fetching and committing offsets. If None, auto-partition assignment (via group coordinator) and offset commits are disabled. Default: None

If you set that to any value, the Consumer will automatically commit the offsets it reads and restart from that position in case it's shutdown.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • I have set it to `group.id=consoleGroup` and on terminal, I was achieving the desired results. Can you tell me how to enable this in the code? `consumer_collect_response = KafkaConsumer('collect-staging-response-latest', bootstrap_servers='localhost:9092', auto_offset_reset='earliest')` So, I have to remove auto_offset_reset='earliest'` and place group_id to some value? Can you tell me what value? – hasherBaba Jun 22 '18 at 09:36
  • Just add `group_id`. And be careful you understand what `auto_offset_reset` means. It's only used in case there are no committed offsets. Once you start using groups and commit it will only be used if your committed offset goes out of range or if they are deleted – Mickael Maison Jun 22 '18 at 11:14
  • I added the group_id and I am not receiving any messages, though they are being inserted into the topic. I ran the producer script for 20 messages. After it was executed, then I ran the consumer script. Also, I put a random name in group_id, is it wrong? However, running consumer script first and then producer script did yield the message in the consumer as well. – hasherBaba Jun 25 '18 at 08:36
  • you can also specify offset to be taken every time a consume is shutdown. – Krishna Oza Jun 28 '18 at 08:33