0

I am trying to send some messages to a Kafka topic in Machine_2 through a python script in Machine_1. Both Machine_2 and Machine_1 are in the same network and both are VM's in Azure.

Code: sampl.py

from kafka import KafkaProducer
Producer = KafkaProducer(bootstrap_servers=['Machine_2:9092'])
Producer.send('test', 'hello')

If I run the above code as

python sampl.py

There is no messages reaching to the Machine_2. However, If I do:

python -i sampl.py

Then the messages reach to the Machine_2. I checked the same using kafka-console-consumer.sh. I did yum update in Machine_1 thinking there might be some libraries missing here. But no luck yet.

Thanks.

wonder
  • 885
  • 1
  • 18
  • 32

1 Answers1

0

kafka-python maintainer here. Producer.send('test', b'hello') is asynchronous and does not provide immediate delivery. What you are likely seeing is that the python interpreter is shutting down before the producer has a chance to complete the network send.

If you want to wait until the message is sent before finishing the script, you should use .get(timeout=...). So try:

Producer.send('test', b'hello').get(timeout=1000)

Or alternately, you can call flush() to do the same for all unsent messages:

Producer.flush(timeout=1000)

dpkp
  • 1,369
  • 7
  • 14
  • What is the default periodic intervals in which the producer sends the messges? What if it doesn't send within the timeout period? – wonder Oct 27 '17 at 03:56
  • kafka-python does not use a periodic interval for sending. Data can be buffered internally to create larger batches for higher throughput, but this is disabled by default. See the docs for more details: http://kafka-python.readthedocs.io/en/master/apidoc/KafkaProducer.html – dpkp Oct 27 '17 at 20:42