0

I an running kafka broker on my windows 10 machine:

[2023-06-19 19:12:33,360] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
3.4.1 (Commit:8a516edc2755df89)

I am running zookeeper and kafka server both using bin\windows\ executables. When I run

kafka-console-producer.bat --topic test --bootstrap-server localhost:9092

And

kafka-console-consumer.bat --topic test --bootstrap-server localhost:9092 --from-beginning

both are working fine and I am able to send messages from producer to consumer.

But when I try to producer using nodejs it gives me below error.

{"level":"WARN","timestamp":"2023-06-19T13:30:50.265Z","logger":"kafkajs","message":"KafkaJS v2.0.0 switched default partitioner. To retain the same partitioning behavior as in previous versions, create the producer with the option \"createPartitioner: Partitioners.LegacyPartitioner\". See the migration guide at https://kafka.js.org/docs/migration-guide-v2.0.0#producer-new-default-partitioner for details. Silence this warning by setting the environment variable \"KAFKAJS_NO_PARTITIONER_WARNING=1\""}
{"level":"ERROR","timestamp":"2023-06-19T13:30:50.304Z","logger":"kafkajs","message":"[Connection] Connection error: connect ECONNREFUSED ::1:9092","broker":"localhost:9092","clientId":"my-app","stack":"Error: connect ECONNREFUSED ::1:9092\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)"}

Code is:

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
})

const producer = kafka.producer()

const run = async () => {
  // Producing
  await producer.connect()
  await producer.send({
    topic: 'test',
    messages: [
      { value: 'Hello KafkaJS user!' },
    ],
  })

  // Consuming
  await consumer.connect()
  await consumer.subscribe({ topic: 'test-topic', fromBeginning: true })

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        value: message.value.toString(),
      })
    },
  })
}

run().catch(console.error);

I tried using kafka-node and kafka npm packages but getting same error.
I have added below lines in server.properties file listeners=PLAINTEXT://localhost:9092 but still no success.

I am expectng a solution which will work with best kafka node package for high scalibility.

I am using this link: [https://www.npmjs.com/package/kafkajs]

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Anand Vaidya
  • 11
  • 1
  • Does the kafkajs consumer work with console producer, and vice-versa? Also, why produce to `test` and consume from `test-topic`? – OneCricketeer Jun 19 '23 at 22:36
  • It is just a typo while I was trying out something. :) But the basic problem still perssits in all the combinations the connection is refused by the broker from nodejs program but the console producer and consumer work smoothly. {"level":"ERROR","timestamp":"2023-06-20T03:44:07.849Z","logger":"kafkajs","message":"[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: connect ECONNREFUSED ::1:9092","retryCount":5,"retryTime":12490} – Anand Vaidya Jun 20 '23 at 03:44
  • I'm able to use KafkaJS fine with a docker instance of Kafka. Are you sure you're not running the node code in WSL2 terminal, since that won't be able to connect to services on windows host? – OneCricketeer Jun 20 '23 at 12:19
  • No, I am running on normal windows command line. All the three zookeeper, kafka server and nodejs program. – Anand Vaidya Jun 21 '23 at 05:46
  • 1) `consumer` is not defined in your question 2) I cannot reproduce the error when it is. You should not need to modify Kafka properties files – OneCricketeer Jun 21 '23 at 18:48
  • Based on the error, though, you might want to check this issue and try disabling IPv6 network adapter and see if that fixes anything https://github.com/tulios/kafkajs/issues/1589 – OneCricketeer Jun 21 '23 at 18:49
  • 1
    THanks a lot this solved the issue. I disabled IPV6 adaptor and also reverted the server.properties file to its original. THanks alot. Closing this thread. I had similar problem with redis I think this can be the same issue. Anyways thanks a lot. – Anand Vaidya Jun 22 '23 at 07:52

1 Answers1

1

I disabled IPV6 adaptor and also reverted the server.properties file to its original. I had similar problem with redis I think this can be the same issue.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Anand Vaidya
  • 11
  • 1