0

I am using kafkajs library to implement kafka consumer in nodejs. I tried to find paramater in kafkajs official document where I could specify number of consumers needed in consumer group, but I couldn't find any.

 const kafka = await kafkaService.instantiateKafkaConnection()
 const kafkaConfig: IKafkaConfig = await kafkaService.getKafkaConfigs()
 const consumer = kafka.consumer({ groupId: kafkaConfig?.consumerConfig?.groupId});
 
 await consumer.connect()
 await consumer.subscribe({ topics: kafkaConfig?.topicConfig?.topic })
        
 await consumer.run({
     eachMessage: async ({ topic, partition, message }) => {
         // business logic
     },
 })

    for(int i=0; i<noOfConsumer; i++){
        const kafka = await kafkaService.instantiateKafkaConnection()
        const kafkaConfig: IKafkaConfig = await kafkaService.getKafkaConfigs()
        const consumer = kafka.consumer({ groupId: kafkaConfig?.consumerConfig?.groupId});

        await consumer.connect()
        await consumer.subscribe({ topics: kafkaConfig?.topicConfig?.topic })
       
        await consumer.run({
            eachMessage: async ({ topic, partition, message }) => {
                // business logic
            }
        })    
    }

The only way I could think of is to loop over same code for creating multiple consumers. Is there any way to pass number of consumer in kafkajs consumer instance? There is parameter partitionsConsumedConcurrently but it does not actually create consumers.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Sukhda Jamidar
  • 11
  • 3
  • Ideally, you'd deploy multiple instances of the application, such as via k8s deployment, or pm2 processes, rather than loop. But what specific problems are you running into here? The loop is fine, but you're overriding the same variables each time – OneCricketeer Sep 29 '22 at 12:54
  • Thanks @OneCricketeer, I was confused with consumer creations within same consumer group that's why I was overriding the variables using loop – Sukhda Jamidar Oct 02 '22 at 12:18

1 Answers1

0

As suggested by @oneCricketeer, we would need to deploy multiple instances of application with the same consumer group Id in order to create multiple consumers.

Sukhda Jamidar
  • 11
  • 3