1

When i try to consume message from topic with more than one partition, consumer not working at all, this is my sample code for initialize kafka and producer:

const kafka = new Kafka({
 clientId: config.kafka.client,
 // eslint-disable-next-line @typescript-eslint/ban-ts-comment
 // @ts-ignore
 brokers: config.kafka.brokers,
 logLevel: logLevel.ERROR,
 retry: {
   retries: 30,
 },
});const admin = kafka.admin();await admin.createTopics({
   topics: [{
     topic: 'topicTest'
   }]
})await admin.createPartitions({
 topicPartitions: [{
   topic: 'topicTest',
   count: 4     // partition count
 }]
})const MyPartitioner = () => {
 // @ts-ignore
 return ({ topic, partitionMetadata, message }) => {
   return message.key % 4;
 }
}const producer = kafka.producer({ createPartitioner: MyPartitioner });
await producer.connect();async sendMessage(topic: string, message: {key: any, value: any}) {
   producer.send({
   topic,
   messages: [{
     key: `${message.key.toString()}`,
     value: JSON.stringify(message.value)
   }]
 }).catch(console.error)
}

And this is how to initialize consumer:

const consumer = kafka.consumer({
   groupId: 'consumer-listener',
   partitionAssigners: [
       roundRobin
   ]
});await consumer.subscribe({
   topic: 'topicTest'
});await consumer.run({
   // autoCommit: false,
   partitionsConsumedConcurrently: 4, // Default: 1
   eachMessage: async ({topic,partition, message}) => {
       console.log(`=========== new message ==========`);
       if(message.key && message.value){
           const key = message.key.toString();
           console.log(partition)
           console.log(key)
       }
      
   }
});

And finally this how to produce some random message:

KafkaClient.sendMessage('topicTest',{
   key: 5,
   value: {
       name: 'a'
   }
});
KafkaClient.sendMessage('topicTest',{
   key: 10,
   value: {
       name: 'c'
   }
});
KafkaClient.sendMessage('topicTest',{
   key: 3,
   value: {
       name: 'b'
   }
});

the messages are produced successfully, but consumer not able to get it. what is wronge with this config? i believe i miss something

output of consumer-group description:

[appuser@broker ~]$ /bin/kafka-consumer-groups --bootstrap-server localhost:9092 --describe -group consumer-listener

GROUP             TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                      HOST            CLIENT-ID
consumer-listener tee             0          -               1               -               mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101  mongerBot-orderBook-fetcher
consumer-listener tee             1          -               1               -               mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101  mongerBot-orderBook-fetcher
consumer-listener tee             2          -               1               -               mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101  mongerBot-orderBook-fetcher
consumer-listener tee             3          -               1               -               mongerBot-orderBook-fetcher-25ef8621-fe5f-4c05-a2d5-77ae6a007f2d /192.168.1.101  mongerBot-orderBook-fetcher

0 Answers0