I have dockerized my project that using kafka for communication, I am using python on consume part of my project, I used two different kafka image because of the NoBrokersAvaible error and I tried all of the combinations I guess. I want to consume messages from kafka but my consumer cannot connect with my kafka brokers.
Console Logs
Traceback (most recent call last):
File "main.py", line 15, in <module>
kafka_consumer = kafka_client.prepare_kafka_consumer()
File "/-----/client/kafka_client.py", line 17, in prepare_kafka_consumer
consumer = KafkaConsumer(self.topic,
File "/usr/local/lib/python3.8/site-packages/kafka/consumer/group.py", line 356, in __init__
self._client = KafkaClient(metrics=self._metrics, **self.config)
File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 244, in __init__
self.config['api_version'] = self.check_version(timeout=check_timeout)
File "/usr/local/lib/python3.8/site-packages/kafka/client_async.py", line 900, in check_version
raise Errors.NoBrokersAvailable()
kafka.errors.NoBrokersAvailable: NoBrokersAvailable
Docker Compose
services:
zookeeper:
image: confluentinc/cp-zookeeper:3.2.0
container_name: zookeeper
hostname: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
restart: always
kafka:
image: confluentinc/cp-kafka:3.2.0
hostname: kafka
container_name: kafka
depends_on:
- zookeeper
ports:
- '9092:9092'
environment:
KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:19091,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
restart: always
app:
build:
context: .
dockerfile: DockerFile
ports:
- "8080:8080"
depends_on:
- kafka
Config.py
class Settings:
KAFKA_BROKERS = ['kafka:19091']
KAFKA_TOPIC = "topic"
MAIL_USERNAME = "mail"
MAIL_PASSWORD = "pass"
settings = Settings()
I am not able to use my app and kafka in same container properly.