When i try to consume message from topic with more than one partition, consumer not working at all, this is my sample code for initialize kafka and producer:
const kafka = new Kafka({
clientId: config.kafka.client,
// eslint-disable-next-line @typescript-eslint/ban-ts-comment
// @ts-ignore
brokers: config.kafka.brokers,
logLevel: logLevel.ERROR,
retry: {
retries: 30,
},
});const admin = kafka.admin();await admin.createTopics({
topics: [{
topic: 'topicTest'
}]
})await admin.createPartitions({
topicPartitions: [{
topic: 'topicTest',
count: 4 // partition count
}]
})const MyPartitioner = () => {
// @ts-ignore
return ({ topic, partitionMetadata, message }) => {
return message.key % 4;
}
}const producer = kafka.producer({ createPartitioner: MyPartitioner });
await producer.connect();async sendMessage(topic: string, message: {key: any, value: any}) {
producer.send({
topic,
messages: [{
key: `${message.key.toString()}`,
value: JSON.stringify(message.value)
}]
}).catch(console.error)
}
And this is how to initialize consumer:
const consumer = kafka.consumer({
groupId: 'consumer-listener',
partitionAssigners: [
roundRobin
]
});await consumer.subscribe({
topic: 'topicTest'
});await consumer.run({
// autoCommit: false,
partitionsConsumedConcurrently: 4, // Default: 1
eachMessage: async ({topic,partition, message}) => {
console.log(`=========== new message ==========`);
if(message.key && message.value){
const key = message.key.toString();
console.log(partition)
console.log(key)
}
}
});
And finally this how to produce some random message:
KafkaClient.sendMessage('topicTest',{
key: 5,
value: {
name: 'a'
}
});
KafkaClient.sendMessage('topicTest',{
key: 10,
value: {
name: 'c'
}
});
KafkaClient.sendMessage('topicTest',{
key: 3,
value: {
name: 'b'
}
});
the messages are produced successfully, but consumer not able to get it. what is wronge with this config? i believe i miss something
output of consumer-group description:
[appuser@broker ~]$ /bin/kafka-consumer-groups --bootstrap-server localhost:9092 --describe -group consumer-listener
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
consumer-listener tee 0 - 1 - mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101 mongerBot-orderBook-fetcher
consumer-listener tee 1 - 1 - mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101 mongerBot-orderBook-fetcher
consumer-listener tee 2 - 1 - mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101 mongerBot-orderBook-fetcher
consumer-listener tee 3 - 1 - mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101 mongerBot-orderBook-fetcher