Stack Overflow
Stack Overflow

Questions tagged [kafkajs]

KafkaJS is an open-source Node.js client library for Apache Kafka

KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.

199 questions
2
votes
1 answer

How to flush data to client just after res.write() is written, before res.end(), in Express.js?

I've created a client of Apache Kafka using Kafkajs, and I'm trying to read messages from a topic in Kafka. It's working fine if I console.log(message). But I want to send message to client everytime whenever a new message is produced/written in the…
S. Joshi
  • 153
  • 1
  • 3
  • 14
2
votes
1 answer

Why is my kafkajs client (Node.js/express.js) throwing 'TypeError: topics.forEach is not a function' while fetching topics metadata?

I'm trying to fetch my kafka brokers' topics' metadata using kafkajs admin client. I've written my server in Node.js + express.js. This is my index.js file, which is the entrypoint of npm. 'use strict'; const express = require('express'); const…
S. Joshi
  • 153
  • 1
  • 3
  • 14
2
votes
0 answers

Is there any way to execute a producer in kafkajs and exit the function without process.exit()?

I'm setting up a server in nodejs that uses kafka. In order to do that I'm using kafkaJS. The problem here is that I don't want the server powering down everytime someone sends a post or reads a value from kafka. Up until now I've tried this to read…
Jose Pinho
  • 33
  • 1
  • 6
1
vote
0 answers

KafkaJS throwing "KafkaJSNumberOfRetriesExceeded: This is not the correct coordinator for this group" with MSK

I have a nestJS application with KafkaJS deployed in an EKS cluster. I am using Amazon MSK as my kafka cluster with three brokers and SASL SCRAM SHA512 Authentication. The Kafka cluster has default configuration. Here is my consumer…
Sampath
  • 11
  • 2
1
vote
1 answer

NestJS Kafka Rebalance & Partition Assignment Strategies, what will happen to already running tasks?

Let's imagine we have: 1 Kafka broker 1 topic, 10 partitions 1 NestJS producer changing number of NestJS consumers (from 1 to 10) After a while, when number of consumers changes, Kafka initiates a rebalancing. AFAIK, NestJS has its own…
DimaisGit
  • 11
  • 2
1
vote
0 answers

The Kafka consumer automatically stops listening for messages after a certain period of time

I am implementing Kafka in Nodejs using node-Kafka/kafkajs. However, I am facing an issue with the consumer. The Kafka consumer stops listening for messages after some time, typically between 1.4 to 2.4 hours. And also we did not get any error or…
Chetan Parmar
  • 31
  • 5
1
vote
0 answers

NestJs Kafka consumer

How can i create kafka consumer with different consumer group ids in one nestjs applicion? Below code works but in one NestJs application i can use only one group-id but I need two consumer with different group-id const app = await…
Elgun Qasimzade
  • 11
  • 1
1
vote
0 answers

Cannot subscribe to topic while consumer is running

I am making a Chat application, using Socket io for sending and receiving messages. I want to use Kafka as a message broker, I am using unique Socket io rooms for each pair of users for privacy, I am expecting the Kafka consumer to subscribe to the…
Karteek Ati
  • 43
  • 5
1
vote
2 answers

In KafkaJS, do I need to pause and resume the consumer when processing a batch?

I use eachBatch() in KafkaJS to process batches of messages. Each batch is written to DB which might take a while under heavy load. I'm worried that the DB might get overloaded when multiple batches are inserted simultaneously. Will KafkaJS call…
aimfeld
  • 2,931
  • 7
  • 32
  • 43
1
vote
2 answers

Kafkajs connection string for service running in same Kubernetes cluster

I have a Nodejs microservice and a Kafka broker running in the same cluster. The kafka broker and zookeeper are running without errors, but I am not sure how to connect to them. kafka.yaml # create namespace apiVersion: v1 kind: Namespace metadata: …
BPDev
  • 397
  • 1
  • 9
1
vote
1 answer

How to stop Kafka retry for specific error

I'm working on a nodeJS/nestJS project and have setup a kafka consumer for topics. I've also added a payload validator that throws an error when the payload in the topic does not meet the required standards. Issue now is that the consumer keeps…
Zephyr
  • 1,612
  • 2
  • 13
  • 37
1
vote
1 answer

Example to make NestJs Kafka work with Avro?

I am looking for an example to use NestJS Kafka microservice work with Avro schema. So far the consumers are working with Json Object but we are looking to towards schema registry with Avro as schema type. Kafka Config: transport:…
JDev
  • 1,662
  • 4
  • 25
  • 55
1
vote
1 answer

Populating kafka messages in an array in Javascript

I am using KafkaJS libaray for consuming messages from a kafka topic. My requirement is to get the messages into an array which I can use in my application later. I can log the messages in the console but not able to populate the array const {…
BIndu_Madhav
  • 577
  • 1
  • 8
  • 21
1
vote
0 answers

How to add consumer and producer credentials in a docker file using jwt authentication

ERROR SASL authentication failed using login context 'Client' with exception: {} The docker is also showing that the token has been verified, but how is autnetication happenng here? `The below given is my yaml file. version: '3.1' services: …
Karthik Nousher
  • 11
  • 1
1
vote
1 answer

How to consume the message from kafka topic count basis or batch model using kafkajs npm

I need to consume the message from Kafka topic. For example, my topic had one thousand records. I need to consume the message count basis. Count as parameter. i.e., if I give count 100 then consume 100 messages only, next loop consumes next 100…
M.S.Udhaya Raj
  • 150
  • 1
  • 12
1 2
3
…
13 14