How can i consume kafka data which is produced in python

Question

I am getting confused in how to create a kafka producer and consumer in pycharm. i have created a produce.py

from time import sleep
from json import dumps
from kafka import KafkaProducer
producer = KafkaProducer(
value_serializer = lambda x:dumps(x).encode('utf-8'),
    bootstrap_servers = ["localhost:9092"]
    )
for i in range(1,100):
    producer.send('test', value = {"hello" : i})
    sleep(0.001)

and a consumer as

from kafka import KafkaConsumer
consumer = KafkaConsumer(
    "test",
     bootstrap_servers = ["localhost:9092"],
     auto_offset_reset = 'earliest',
     enable_auto_commit = True,
     group_id = None,
     )

for message in consumer:
    print(message)

there is no error showing both the producer and consumer are running but my consumer is not giving an output.

score 1 · Answer 1 · answered Oct 01 '21 at 14:19

Start your consumer before the producer to ensure it'll read all messages (change/reset the group id if you have to). GroupId shouldn't be none if you want to auto commit

Call producer.flush() after the for loop. Otherwise, you're going to have a pending batch that's removed from memory when the producer script ends, and will not be sent to the broker

You can also use Kafka console consumer tools if you think that the Python consumer is the issue

How can i consume kafka data which is produced in python

1 Answers1