1

I am implementing Kafka in Nodejs using node-Kafka/kafkajs. However, I am facing an issue with the consumer. The Kafka consumer stops listening for messages after some time, typically between 1.4 to 2.4 hours. And also we did not get any error or event related to it.

Currently, there is no database operation performed.

I have used docker for Kafka on Ubuntu.

Below is Kafka configuration

const consumerOptions = {
  groupId: 'TESTDATA',
  fetchMinBytes: 1 * 1024, // Minimum of 1 KB
  fetchMaxBytes: 10 * 1024 * 1024, // Maximum of 5 MB
  heartbeatInterval: 5000, // Heartbeat every 3 seconds
  rebalanceTimeout: 60000, // Allow up to 60 seconds for rebalance
  protocol: ['roundrobin'],
  fromOffset: 'latest',
  encoding: 'utf8',
  keyEncoding: 'utf8',
  valueEncoding: 'utf8',
  keyDeserializer: kafka.KeyDeserializer, // Corrected keyDeserializer import
  valueDeserializer: jsonDeserializer,
  fetchMaxWaitMs: 5000,
  fetchErrorHandling: 'commit', // Commit offsets for partitions with fetch errors
  maxMessages: 1000, // Retrieve up to 1000 messages per fetch request
  autoCommit: true,             // Enable auto-commit
  autoCommitIntervalMs: 5000,   // Auto-commit interval (in milliseconds)
  sessionTimeout: 30000
};
Chetan Parmar
  • 31
  • 5
  • I've had kafkajs consumers running fine for months. The library does not disconnect on its own. Instrument your consumer and add more logs - https://kafka.js.org/docs/instrumentation-events – OneCricketeer Jun 30 '23 at 20:31
  • can you provide more details? How did you find that the consumer was not listening? Is there any crash in the server? The more information you provide, the better chance of getting a solution. – Amit Tiwary Jul 03 '23 at 05:38
  • Generally, I put console.log in the consumer and observe them. After some time, the consumer automatically stops receiving messages, and then the logs cannot be printed. There are no crashes in the server, all services are working properly, and there are no errors or events generated. the only massage did not receive by the consumer. – Chetan Parmar Jul 03 '23 at 10:02
  • I got an error in the Zookeeper server mentioned below java.io.IOException: Len error. A message from /87.251.64.11:64984 with advertised length of 302055518 is either a malformed message or too large to process (length is greater than jute.maxbuffer=1048575) at org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:557) at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:347) – Chetan Parmar Jul 04 '23 at 11:57

0 Answers0