Stack Overflow
Stack Overflow

Questions tagged [kafkajs]

KafkaJS is an open-source Node.js client library for Apache Kafka

KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.

199 questions
3
votes
0 answers

What does this kafka error mean on a shared cluster setup and how to resolve this?

I have been using the shared cluster setup for some time now to test some stuff on CloudKarafka. It gave me no problems until yesterday when I started getting this error message.…
maopuppets
  • 420
  • 3
  • 9
  • 27
2
votes
1 answer

Take to long for kafkaJS to connect

I'm recently using nestJS and kafkaJs for my microservice project and after each server refresh (debug mode) the KafkaJS take too long to connect to the kafka cluster again, is there some configuration missing? My kafkaJS version is 2.2.3 and I have…
mmRoshani
  • 99
  • 7
2
votes
0 answers

Client network socket disconnected before secure TLS connection was established. How can I connect to a kafka cluster using Kafka JS in Node js?

The credentials that I am provided with :- (I also have kafka.keystore.jks and kafka.truststore.jks) host: xxxxx-xxxxx-x.cloudclusters.net port: xxxxx ip: xxx.xxx.xxx.xx trustore pw: xxxxxxxx keystore pw: xxxxxxxx I think I am not using all the…
Abdullah Ch
  • 1,678
  • 1
  • 13
  • 31
2
votes
0 answers

kafkajs "The coordinator is not aware of this member" error

I am trying to add consumers for a topic using kafkajs (v2.0.0) on my node project. i am trying to add a consumer, using example provided in the docs creating the client: const kafka = new Kafka({ logLevel: logLevel.DEBUG, brokers:…
harelos
  • 21
  • 3
2
votes
0 answers

what's the purpose of `producer.disconnect or `consumer.disconnect` in kafkajs?

what's the purpose of the disconnect functions in kafkajs? const producer = kafka.producer() await producer.connect() await producer.send({ topic: 'test-topic', messages: [ { value: 'Hello KafkaJS user!' }, ], }) await…
Uri
  • 25,622
  • 10
  • 45
  • 72
2
votes
0 answers

KafkaJs Crash: Value should be an instance of Encoder

I have 6 partitions in a topic and I have a nestjs application running on Kubernetes with 6 replicas , Idea is to assign each 1 partition to 1 replica . Whenever I push a new build , I get below error from 3 out of 6 replicas , while other 3 gets…
Dosank
  • 41
  • 2
2
votes
2 answers

How do you consume the latest message from Kafka topic?

I'm using KaafkaJS to connect and consume the kafka messsages. I'm using the following code to consume messages. But it fetches all the messages but I would like to store the last message in the variable messageValue. Can someone help me with it…
Bijo Joseph
  • 31
  • 1
2
votes
0 answers

how to read x number of messages in kafkajs consumer at a time

i have situation where to achieve better performance i have to read multiple kakfka message at a time, i have search on the internet and found their is functionality of kafka called batch where we can read messages in batch the problem is that i am…
mohit
  • 91
  • 6
2
votes
1 answer

NestJS-Kafka group id unchanged after I updated it in the code

I'm using the Kafka JS to connect to kafka. In my consumer, initially I named my group id as 'orderable-test', later when I updated the this group id to 'orderable-test-1' and rerun my code, the group id remains the 'orderable-test'. Even if I…
Zichzheng
  • 1,090
  • 7
  • 25
2
votes
1 answer

Can't send Kafka messages though topic and producer both exist

I'm working in TypeScript with the KafkaJS library locally, with a single kafka broker. I've connected a producer successfully, have verified that my topic was created, and am generating messages with: const changeMessage = { key: id, …
mojones101
  • 161
  • 1
  • 2
  • 12
2
votes
1 answer

unable to retrieve kafka confluent schema regsitry registry via `@kafkajs/confluent-schema-registry`

I'm using @kafkajs/confluent-schema-registry in a typescript node app, attempting to pull down schemas. I have done this in Java with the analogus confluent schema registry java library w\o issue. When creating the schema registry client, I don't…
Stealth Rabbi
  • 10,156
  • 22
  • 100
  • 176
2
votes
1 answer

KafkaJS: ECONNREFUSED when trying to produce a message on a topic

I'm using KafkaJS to produce a message on a Kafka topic. To do so, I've put the Kafka server in Docker using the wurstmeister image. What I want to do: the Poll container produce a message to the Poll topic and consume messages from the responsePoll…
Couldosh
  • 371
  • 3
  • 17
2
votes
0 answers

kafkajs consumer lag, but messages commited

need some help with kafkajs consumer lag one kubernetes pod send 10 messages to queque, last offset in kafdrop is 10, all messages are succesfully delivered to 3rd party system and succesfully, commited but, and its random, there is consumer lag in…
Ondrej Basista
  • 21
  • 4
2
votes
1 answer

kafkajs - gracefully stop kafkajs instance after disconnect

I'm using kafkajs on both my production and integration tests. Before all my tests I'm creating a kafkajs instance with producer & consumer connect/subscribe/run(eachMeassage)... After all my tests I want to stop gracefully all my node process…
Benjamin Bohec
  • 118
  • 2
  • 11
2
votes
1 answer

How to pause Nest.js Kafka consumer

I am using Kafka.js in a Nest.js project. This is how I initialise KafkaClient: @Module({ ... providers: [{ provide: 'KAFKA_CLIENT', useFactory: async (configService: KafkaClientConfigService) => { const kafkaOptions =…
Yuvals
  • 3,094
  • 5
  • 32
  • 60
1
2
3
…
13 14