0

I am publishing messages via golang using segmentio-go. And I want to read those message using python for which I tried kafka-python, pykafka.
In all these libraries, I cannot receive the message, I am able to read the messages when reading it from golang by creating a consumer at golang side. And also when I am publishing messages via python, I am able to read it from the python consumer as well as from golang consumer, but those topics in which messages are published via golang, consumer in python side gets completely hanged/stuck, code doesn't crash but also don't respond.

In the python consumer, I am able to get broker and its topics, It is only for those topics in which messages are published via golang, python consumer gets hanged.

Code for Python-consumer:

consumer = KafkaConsumer(<topic name>, bootstrap_servers=[<ip:port>], auto_offset_reset='earliest', group_id=None, max_partition_fetch_bytes=104857600) 
print(consumer.topics()) # Able to get all topic names

for message in consumer:
    print(message.value)

Code for Golang Producer

l := log.New(os.Stdout, "kafka framer: ", 0)

w := kafka.NewWriter(kafka.WriterConfig{
   Brokers: []string{broker_address},
   Topic:   topic_name,
   Logger: l,
})

err := w.WriteMessages(ctx, kafka.Message{
        Key: []byte(strconv.Itoa(0)),
        Value: []byte(data),
})

if err != nil {
   panic("could not write message " + err.Error())
}

Please help.

Mohammad Nayeem
  • 1
  • It's not hanging if there are actually no messages. Sounds like the golang producer isn't flushing the buffer... Have you tried adding `defer w.Close()` like the examples have? – OneCricketeer May 11 '21 at 22:42
  • @OneCricketeer I tried defer w.Close(), still no luck. I am able to access the messages via golang consumer, it is when I am trying to access it via python it is not receiving messages – Mohammad Nayeem May 13 '21 at 05:28
  • Maybe you should remove `group_id=None` – OneCricketeer May 13 '21 at 12:29
  • 1
    Thanx, I got it resolved. Actually I had to receive messages from another remote server, for which I had to advertise the ip addredd of broker from the local machine. [This](https://stackoverflow.com/questions/27191347/why-i-cannot-connect-to-kafka-from-outside) help me in resolving it. – Mohammad Nayeem May 16 '21 at 19:52

0 Answers0