Im running a Kafka Broker Cluster containing 3 brokers. I connected my Nodejs application with cluster. When all three brokers are up, application is connected. everything works fine. but when i shutdown the leader broker in the cluster. then application retries to connect with broker in cluster. but i continously get the error, 'There is no leader for this topic-partition as we are in the middle of a leadership election.'
That is my producer application code.
console.log("producer..........")
const { Kafka } = require('kafkajs')
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['localhost:8092', 'localhost:8093', 'localhost:8094']
})
const producer = kafka.producer()
// const consumer = kafka.consumer({ groupId: 'test-group' })
const run = async () => {
// Producing
await producer.connect()
setInterval(async ()=> {
await producer.send({
topic: 'test-topic',
messages: [
{ value: 'Hello KafkaJS user!' },
],
})
}, 2000)
}
run().catch(console.error)
and im running brokers on docker. that is my docker-compose file.
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper
container_name: zookeeper
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
networks:
- kafka-network
kafka1:
image: confluentinc/cp-kafka
container_name: kafka1
ports:
- 8092:8092
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,CONNECTIONS_FROM_HOST://localhost:8092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
volumes:
- kafka1-data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- kafka-network
kafka2:
image: confluentinc/cp-kafka
container_name: kafka2
ports:
- 8093:8093
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093,CONNECTIONS_FROM_HOST://localhost:8093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
volumes:
- kafka2-data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- kafka-network
kafka3:
image: confluentinc/cp-kafka
container_name: kafka3
ports:
- 8094:8094
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094,CONNECTIONS_FROM_HOST://localhost:8094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_BROKER_ID: 3
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
volumes:
- kafka3-data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- kafka-network
volumes:
kafka1-data:
kafka2-data:
kafka3-data:
networks:
kafka-network:
