3

I am attempting to send data to a Kafka topic via python as a remote producer. My script reports no exceptions, but nothing shows up in Kafka. I'm able to get a response from the brokers (all on one separate host) using the consumer method. Looking through the forums I saw to make sure and flush the write cache, but no luck there. Script is below:

from kafka import KafkaProducer, KafkaConsumer
from json import dumps
producer = KafkaProducer(bootstrap_servers='192.168.1.100:9093', value_serializer=lambda x: dumps(x).encode('utf-8'))
producer.send('home-sensehat-temperature',{"timestamp": "2020-08-12 23:31:19.102347", "temperature": 127.6969})
producer.flush()
consumer=KafkaConsumer(bootstrap_servers='192.168.1.100:9093')
print(consumer.topics())

The response I get from consumer.topics() is: {'home-sensehat-temperature', 'home-camera-path', 'home-sensehat-humidity', 'home-sensehat-pressure'}
So this implies I can make a good connection to the brokers.

I tried digging through the kafka broker logs but couldn't find anything. Any help would be greatly appreciated!

Dav25557
  • 31
  • 4
  • I should also add that I'm able to write to this same topic via Python on the broker host itself – Dav25557 Aug 14 '20 at 04:08
  • Do you make sure your client has conencted to kafka, kafka client has buffer, even if connected failed, you also can publish message which will be buffered. – spike 王建 Aug 14 '20 at 04:35
  • this page is related to connecting issues, https://stackoverflow.com/questions/27191347/why-i-cannot-connect-to-kafka-from-outside . – spike 王建 Aug 14 '20 at 04:38
  • Thanks @Spikie. My reasoning is that because I get a good response from consumer.topics(), I have a good connection to the brokers. Is there something else I should do to verify I have connectivity? – Dav25557 Aug 14 '20 at 15:27
  • What do you mean "not getting to Kafka"... Did you check the offsets of the topic? – OneCricketeer Aug 14 '20 at 16:26
  • Hi @OneCricketeer, I check by starting a consumer in the CLI and watching for the message to come in. Retention policy is one day, so it's not like the messages are immediately removed before I can see them. That answer what you're asking? – Dav25557 Aug 15 '20 at 02:50
  • 1
    I also faced with same problem. It's solved for me by moving producer to same host with kafka and connect to localhost – Slava Rozhnev Aug 15 '20 at 07:37
  • Thanks @SlavaRozhnev, As a similar option, I'm considering putting a separate instance of Kafka on this device, and using the Kafka connector. It would have the benefit of a persistent buffer, but since I'm working on a raspberry pi Zero, it may be too resource intensive. Thanks for the input! – Dav25557 Aug 15 '20 at 16:17
  • I meant use GetOffsetShell command to check offsets. But yes, rasp pi doesn't really have enough ram to run things – OneCricketeer Aug 16 '20 at 16:52

1 Answers1

0

Got it! Everything was fine on the remote producer, but I had to add a config parameter on the brokers. Specifically the advertised.listeners parameter in the config/server.properties file that each broker uses. Details are in the below post:

Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?

Thanks everyone!

Dav25557
  • 31
  • 4