2

First I would like to say that I am a newbie to Kafka and also stackoverflow, So I am sorry if I am not asking this in the right way. I am trying to implement the producer-consumer using kafka-python. But its not working properly

I have the zookeeper installed and its up and running. I have up the kafka-server also. But when I am running the consumer and producer through pycharm, the messages are not getting received by the receiver.The consumer keeps on running but the producer stops.

consumer.py

from kafka import KafkaConsumer

consumer = KafkaConsumer('test', group_id='test-consumer-group', 
bootstrap_servers=['my_ip:9092'], api_version=(0, 10, 1),
                     auto_offset_reset='earliest')
print("Consuming messages")
for msg in consumer:
    print(msg)

producer.py

from kafka import KafkaProducer

print('above producer')
producer = KafkaProducer(bootstrap_servers=['my_ip:9092'], api_version=(0, 10, 1),
                         compression_type=None
                         )
print('after producer')
for _ in range(100):
    producer.send('test', b'HELLO NITHIN chandran')

print('after sending messages')

In the place of my_ip, I have provided with my system ip address from ipconfig.

consumer.py Output -

Consuming messages

The consumer.py doesnt stop running

producer.py Output -

above producer
after producer
after sending messages

Process finished with exit code 0

The producer.py stops running and the process is finished as shown in the output.

Please help me in resolving this issue. All help are appreciated

Stan11
  • 274
  • 3
  • 11
  • Welcome to Stackoverflow, Can you check the topic's data in data dir of Kafka? Please check that there is anything in the topic or not? – Amin Aug 27 '19 at 05:31
  • How do I do that? There is no data directory in Kafka. There are only bin,config,libs,site-docs,LICENSE,NOTICE. – Stan11 Aug 27 '19 at 05:34
  • you should look at the config file in `config/server.properties`. probably it is in the `/tmp` directory. – Amin Aug 27 '19 at 05:52
  • Could u please be a bit more specific. Like what should I be looking for in the ```config/server.properties``` file – Stan11 Aug 27 '19 at 05:59
  • Of course. there is a `log.dirs` config, in this file. it specifies the directory of storing topics' data. in that directory, there are many folders with your topics name. find your topic and open the log file. – Amin Aug 27 '19 at 06:04
  • Yes there are 4 files but none with my topic name. There is one LOCK file also. But all the files are empty the size is 0KB. – Stan11 Aug 27 '19 at 06:22
  • ```https://www.learningjournal.guru/article/kafka/installing-kafka-on-windows/``` – Stan11 Aug 27 '19 at 06:23
  • I had set everything up through the above mentioned link, this is through the command prompt – Stan11 Aug 27 '19 at 06:24
  • When I run the consumer through the command prompt, I get the below mentioned warning now – Stan11 Aug 27 '19 at 06:27
  • `WARN [Consumer clientId=consumer-1, groupId=console-consumer-52241] 1 partitions have leader brokers without a matching listener, including [test-0] (org.apache.kafka.clients.NetworkClient)` – Stan11 Aug 27 '19 at 06:27
  • Probably you have problem with `advertised.listeners`. set Kafka address to localhost instead of your IP – Amin Aug 27 '19 at 07:00
  • Yes I have used localhost only – Stan11 Aug 27 '19 at 07:01
  • I have used localhost for both `advertised.listeners` and `listeners` but still the issue persists – Stan11 Aug 27 '19 at 07:11
  • Please follow the conversation here https://chat.stackoverflow.com/rooms/198510/kafka-issues – Amin Aug 27 '19 at 07:18
  • It says I must have 20 reputations to talk in that – Stan11 Aug 27 '19 at 07:20
  • but i am not able to type anything in there – Stan11 Aug 27 '19 at 07:32
  • Sorry, these comments are being conversational. I can talk to you about your problem in another place. please suggest somewhere. – Amin Aug 27 '19 at 07:36
  • Yaa sure Is LinkedIn okay? – Stan11 Aug 27 '19 at 07:40
  • or where do all normally do it? – Stan11 Aug 27 '19 at 07:40

1 Answers1

1

Your code is ok, the problem is about your broker configuration. Please set it to initial configuration, just change the log.dirs to the path that you want to store Kafka data. After changing the config file follow these steps:

  • Stop zookeeper and kafka
  • Clear both kafka and zookeeper data dir
  • Run zookeeper and kafa
  • Start consumer and producer
Amin
  • 975
  • 8
  • 24