Facing connection issues within confluent-kafka in docker compose environment

Question

I’m relatively new to Kafka and currently trying to set up a Kafka consumer within a Docker Compose environment. Unfortunately, I’ve hit a roadblock with connection issues and I’d greatly appreciate your expertise in helping me troubleshoot this problem.

Issue: I’m encountering a “Connection refused” error when attempting to establish a connection between a Kafka consumer and a Kafka broker within Docker containers. The error message, which includes details of the failed connection attempt, is as follows:

notification-service-notification-service-1 | %3|1693292196.192|FAIL|rdkafka#consumer-1| [thrd:sasl_plaintext://kafka:9092/bootstrap]: sasl_plaintext://kafka:9092/bootstrap: Connect to ipv4#172.28.0.4:9092 failed: Connection refused (after 8ms in state CONNECT)

Setup:

I’m using Docker Compose to manage my services, including Kafka and ZooKeeper.
The Kafka consumer is implemented using the confluent-kafka library to consume messages from a Kafka topic.
My Kafka broker and ZooKeeper are configured using environment variables.

this is my Kafka consumer

class KafkaConsumer:
    def __init__(self, topic):
        config = get_config()

        self.consumer_config = {
            "bootstrap.servers": config.kafka_servers,
            "group.id": config.kafka_consumer_group_id,
            "security.protocol": config.kafka_security_protocol,
            "sasl.mechanism": config.kafka_sasl_mechanisms,
            "sasl.username": config.kafka_username,
            "sasl.password": config.kafka_password,
            "auto.offset.reset": "earliest",
        }

        self.consumer = Consumer(self.consumer_config)
        self.topic = topic

    def deserializer(self, serialized):
        try:
            json_data = json.loads(serialized)
            notification = Notification(**json_data)
        except Exception as e:
            logger.error(e)
            return

        return notification

    async def start(self):
        self.consumer.subscribe([self.topic])
        try:
            while True:
                msg = self.consumer.poll(1.0)
                if msg is None:
                    continue
                if msg.error():
                    if msg.error().code() == KafkaError._PARTITION_EOF:
                        continue
                    else:
                        logger.error(msg.error())
                else:
                    notification = self.deserializer(msg.value())
                    if notification:
                        try:
                            trigger_push_notification(notification)
                        except Exception as e:
                            logger.error(e)
        finally:
            self.consumer.close()

and here is env var

KAFKA_SERVERS=kafka:9092
KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT
KAFKA_SASL_MECHANISMS=PLAIN