KafkaJS consumer disconnects after connecting to Kafka container

Question

I have implemented a typescript application containing the "Kafka JS" library to initialize a Kafka consumer within it.

This is the code of the consumer:

export class KafkaConsumer {
  public kafka: Kafka = {} as Kafka;
  public consumer: Consumer = {} as Consumer;

  public async InitKafkaConsumer() {
    this.kafka = new Kafka({
          logLevel: logLevel.DEBUG,
          clientId: 'kafka-consumer-client',
          brokers: ['kafka:29092'],
          ssl: false,
          retry: {
            initialRetryTime: 100,
            retries: 8
          }
        } as KafkaConfig);
    
    this.consumer = KafkaConsumer.kafka.consumer({
        groupId: 'kafka-consumer-group',
        sessionTimeout: 300000
    });
    await this.consumer.connect();
  }
}

In addition, I composed a single Kafka broker with all its dependencies, both are deployed in a docker-compose file:

myservice:
    build: 
      context: .
      dockerfile: Dockerfile
    ports:
      - 8086:8086
    depends_on:
      - kafka
    networks:
      - kafka-network
zookeeper:
    image: "confluentinc/cp-zookeeper"
    container_name: zookeeper
    restart: always
    ports:
      - 2181:2181
    environment:
      TZ: ${TZ_SET}
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - kafka-network
kafka:
    image: "confluentinc/cp-kafka"
    container_name: kafka
    restart: always
    depends_on:
      - zookeeper
    ports:
      - '9092:9092'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 100
      CONFLUENT_METRICS_ENABLE: 'false'
    networks:
      - kafka-network

networks:
  kafka-network:

Eventually, while running the compose file and the application started to ran, the consumer can't connect to the broker.

These are the logs from the container:

2023-01-02 13:22:04 {"level":"ERROR","timestamp":"2023-01-02T11:22:04.032Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:04 {"level":"ERROR","timestamp":"2023-01-02T11:22:04.033Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":0,"retryTime":90}
2023-01-02 13:22:05 {"level":"ERROR","timestamp":"2023-01-02T11:22:05.126Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:05 {"level":"ERROR","timestamp":"2023-01-02T11:22:05.127Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":1,"retryTime":174}
2023-01-02 13:22:06 {"level":"ERROR","timestamp":"2023-01-02T11:22:06.303Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:06 {"level":"ERROR","timestamp":"2023-01-02T11:22:06.304Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":2,"retryTime":400}
2023-01-02 13:22:07 {"level":"ERROR","timestamp":"2023-01-02T11:22:07.705Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:07 {"level":"ERROR","timestamp":"2023-01-02T11:22:07.706Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":3,"retryTime":734}
2023-01-02 13:22:09 {"level":"ERROR","timestamp":"2023-01-02T11:22:09.442Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:09 {"level":"ERROR","timestamp":"2023-01-02T11:22:09.443Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":4,"retryTime":1448}
2023-01-02 13:22:11 {"level":"ERROR","timestamp":"2023-01-02T11:22:11.892Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:11 {"level":"ERROR","timestamp":"2023-01-02T11:22:11.893Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":5,"retryTime":2584}
2023-01-02 13:22:15 {"level":"ERROR","timestamp":"2023-01-02T11:22:15.481Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:15 {"level":"ERROR","timestamp":"2023-01-02T11:22:15.483Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":6,"retryTime":4660}
2023-01-02 13:22:21 {"level":"ERROR","timestamp":"2023-01-02T11:22:21.145Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:21 {"level":"ERROR","timestamp":"2023-01-02T11:22:21.149Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":7,"retryTime":10256}
2023-01-02 13:22:32 {"level":"ERROR","timestamp":"2023-01-02T11:22:32.407Z","logger":"kafkajs","message":"[Connection] Connection timeout","broker":"kafka:29092","clientId":"kafka-consumer-client"}
2023-01-02 13:22:32 {"level":"ERROR","timestamp":"2023-01-02T11:22:32.409Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection timeout","retryCount":8,"retryTime":19094}
2023-01-02 13:22:32 {"level":"error","message":"KafkaConsumner >> StartConsuming >> failed with error: [KafkaJSNumberOfRetriesExceeded: Connection timeout]"}
2023-01-02 13:22:32 node:internal/process/promises:246
2023-01-02 13:22:32           triggerUncaughtException(err, true /* fromPromise */);
2023-01-02 13:22:32           ^
2023-01-02 13:22:32 
2023-01-02 13:22:32 KafkaJSNonRetriableError
2023-01-02 13:22:32   Caused by: KafkaJSConnectionError: Connection timeout
2023-01-02 13:22:32     at Timeout.onTimeout [as _onTimeout] (/node-typescript-template/node_modules/kafkajs/src/network/connection.js:223:23)
2023-01-02 13:22:32     at listOnTimeout (node:internal/timers:557:17)
2023-01-02 13:22:32     at processTimers (node:internal/timers:500:7) {
2023-01-02 13:22:32   name: 'KafkaJSNumberOfRetriesExceeded',
2023-01-02 13:22:32   retriable: false,
2023-01-02 13:22:32   helpUrl: undefined,
2023-01-02 13:22:32   retryCount: 8,
2023-01-02 13:22:32   retryTime: 19094
2023-01-02 13:22:32 }

What could be the problem?

Thanks in advance!

Are you sure Kafka is actually starting? `depends_on` doesn't wait for the actual container to be healthy... Don't try to connect immediately — OneCricketeer, Jan 02 '23 at 17:13
Basically, I searched for the best solution to add a health check for Kafka container, but without something clear and helpful enough. — Or S, Jan 02 '23 at 17:56
You can use AdminClient from KafkaJS. Otherwise, just add like 30 second sleep before starting consumer — OneCricketeer, Jan 02 '23 at 18:03

KafkaJS consumer disconnects after connecting to Kafka container

0 Answers0