Kafka on Docker for Windows - cannot consume data

Question

I created a kafka setup using bitnami/kafka docker template. After recreating the containers multiple times (I wasn't able to connect to Kafka on 9092) I finally got my cluster up and running. I believe the problems were caused by the fact that Docker is intalled on Windows and not Linux.

My current problem is that I'm not able to consume any messages using the kafka-python library and the following piece of code:

from kafka import KafkaConsumer
from kafka import TopicPartition

print('Making connection.')
consumer = KafkaConsumer(bootstrap_servers='localhost:9092')

print('Assigning Topic.')
consumer.assign([TopicPartition('linuxhint', 2)])

print('Getting message.')
for message in consumer:
    print("OFFSET: " + str(message[0])+ "\t MSG: " + str(message))

The code doesn't fail. It prints the "Getting message" line and doesn't finish. I tried consuming the same topic using the Conduktor app, and I can't do it either. Conduktor isn't even able to read the count and size of the topic. How can I troubleshoot it?

Answer 1

To troubleshoot, exec into the Kafka container and run

kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic linuxhint

Then outside the container, make sure to configure the KAFKA_CFG_ADVERTISED_LISTENERS appropriately (this property is only for bitnami containers) and expose the correct ports in your Docker command.

https://github.com/bitnami/bitnami-docker-kafka#accessing-kafka-with-internal-and-external-clients

Then, if you're using WSL2 docker integration, but not running Python within the hypervisor (i.e. instead running the code from windows), like what Conduktor would do, then I think there's some additional port forwarding that needs to be done outside of Docker. Therefore, next debug step would be to run the code from WSL2 directly.

And finally make sure that you're using the earliest offsets of your consumer group and/or start the consumer after you run a producer (any producer code you're running would have similar connectivity issues)