0

I am using Nodejs package for Kafka consumer. I have 100 topic each having 1 partition.

const { Kafka, logLevel } = require("kafkajs");
const kafka = new Kafka({
   logLevel: logLevel.INFO,
   clientId: "kafka9845",
   brokers:["90.45.78.123"],
   connectionTimeout:30000,
   sessionTimeout:30000,
   requestTimeout:30000,
   heartbeatInterval:10000,
   retry: {
      initialRetryTime:5000,
      retries:200
   }
});
const topicGroups = ["topic1","topic2",....];
const kafkaConsumer = kafka.consumer({groupId: 'kafkaConsumer', fromBegining:true});
const funKafkaConsumer = async ( kafkaConsumer ) => {
   await kafkaConsumer.subscribe({ topics: topicGroups });
   await kafkaConsumer.run({
    autoCommit: false,
    eachMessage: async (task) => {
        console.log(task);
        await kafkaConsumer.commitOffsets([{topic: task.topic, offset : (Number(task.message.offset) +1).toString()}])
    }
  });
}

kafkaConsumer.on('consumer.crash', async (payload) => {
    try {
      kafkaConsumer.disconnect();
    } catch(error) {

   } finally {
    setTimeout( async () => {
        await kafkaConsumer.connect();
        funKafkaConsumer(kafkaConsumer).catch(console.error);
    }, 5000);
   }
});
 
const funConnect = () => {
   kafkaConsumer.connect();
   funKafkaConsumer(kafkaConsumer).catch(console.error);
}

funConnect();
process.on('SIGINT', function () {
  kafkaConsumer.disconnect();
});

I am using "kafkajs":"^2.0.0" and "kafka-node":"^5.0.0"

Every 3 minute we are producing message inside different topics

Problem: I have analysis message consumer processing in queue way.

Requirement: How can make parallel message consuming.

Sudhir
  • 835
  • 11
  • 31
  • This code looks fine... 1) You don't need two different Kafka libraries. 2) Have you read the documentation on concurrency setting? – OneCricketeer Jul 18 '23 at 12:48
  • @OneCricketeer I am using same code. But some message taking time to consume. Is their is any way we can priority the topic – Sudhir Jul 18 '23 at 17:56
  • 1) Producers do not "send" immediately, they buffer data in memory first. Could that be your issue? 2) NodeJS has a single event loop, and will not parallelize hundreds of topics very well. A more scalable approach would be hundred unique NodeJS processes, maybe accepting some `--topic=X` CLI flag, similar to `kafka-console-consumer` – OneCricketeer Jul 18 '23 at 18:58
  • @OneCricketeer Thanks for your response. To handle this can we create 100 consumer for each topic dynamically. I have one doubt how much minimum memory and CPU utilization need if I have create dynamic consumer – Sudhir Jul 19 '23 at 12:25
  • I don't have personal experience with that, but a modern computer with quad/octa core can easily handle 100 processes at the same time. Then based on my current KafkaJS apps, the memory usage is around 120MB, so 12GB of memory at least, plus overhead, so 16GB of RAM, at least on one machine... Obviously if you use a cluster like Kubernetes or AWS ECS, then you can distribute to many machines, where each need less resources – OneCricketeer Jul 19 '23 at 20:19

0 Answers0