Stack Overflow
Stack Overflow

Questions tagged [kafkajs]

KafkaJS is an open-source Node.js client library for Apache Kafka

KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.

199 questions
0
votes
1 answer

Kafkajs run multiple consumer in single file

I am using kubernetes microserice in which I have create single Kafka micro service to consume all topic and perform its activities. Code sample below: const { Kafka, logLevel } = require("kafkajs"); const kafka = new Kafka({ logLevel:…
Sudhir
  • 835
  • 11
  • 31
0
votes
0 answers

NestJS - Kafka unhandled exceptions can't be caught

I've been trying to catch exceptions coming from a kafka microservice but none of the proposed solutions I've found work The error is coming from here: server-kafka.js async listen(callback) { try { this.client =…
user17714968
  • 25
  • 5
0
votes
1 answer

Implementing Single Partition Assigner on Kafkajs

I am trying to assign a consumer on single partition of a topic using kafkajs client library in TypeScript. I am using partitionAssigners in the consumer group as provided in documentation. I have created my SinglePartitionAssigner but it doesn't…
Ashutosh Pandey
  • 120
  • 8
0
votes
1 answer

Least permission set to be able to use (nodejs) kafkajs's AdminAPI.listTopics()

The popular NodeJS Kafka client, kafkajs, has a method on the admin client part of it to list all the topics in the broker. https://kafka.js.org/docs/admin However this method returns an empty array when the Kafka user doesn't have enough privileges…
tidra
  • 1
0
votes
2 answers

Custom Consumer Assignment Strategy Implementation in TypeScript

I have a service, which is running in Kubernetes, During peak time, this service can scale in 1000s. Each service instance is consuming events from Kafka, We are using the Managed Streaming Kafka(AWS MSK) as broker, Since topics are going to be so…
Ashutosh Pandey
  • 120
  • 8
0
votes
1 answer

Kafka - Registering to topics matching regex, not always existing (yet)

I run an IoT platform that relies on MQTT (my broker is aedes) for some features. To scale my backend better, I started implementing a Kafka service. The devices will speak with MQTT, which will relay messages to Kafka, and the backend will…
braoutch
  • 13
  • 4
0
votes
1 answer

How can I read the same message several times in KafkaJs?

Here is the thing - I have to read a message from Kafka and check if some specific header has date >= than now ---> then handle it accordingly, otherwise I should be able to get the same message as many times as needed for the condition above to…
Jun
  • 21
  • 5
0
votes
0 answers

Looking for approach to minify kafkajs as a JS library instead of NPM

I am looking for minified/dist version in order to include directly in our packaged product as a library due to limitations with npm on a SaaS platform. Has someone used/created a dist version of Kafkajs code? Not able to find a minified version of…
DCom Dev
  • 3
  • 1
0
votes
0 answers

How to properly E2E test production | consumption of messages to Kafka with NestJS and Jest

I want to be able to test from e2e the production and consumption of messages from a NestJS application to a broker. In my case I have put up a public repository here with the problem in total. The application uses cp-all-in-one from confluent for…
brasidas
  • 1
  • 1
0
votes
0 answers

Kafkajs. Lot of error logs being generated and how to handle it

I have been noticing or a while, a lot of error logs being generated from kafkajs. Sometimes it just doesn't stop stop. Some of the example of the logs are the following. Log1 ,\"logger\":\"kafkajs\",\"message\":\"[Connection] Connection error:…
Missak Boyajian
  • 1,965
  • 7
  • 32
  • 59
0
votes
1 answer

NestJS: Kafka client - disable consumer

I'm trying to produce messages in NestJS with kafka, but it seems the default of the ClientsModule is to start consuming a topic and I see no way in the docs to disable that. Created a client using: return ClientProxyFactory.create({ …
user17714968
  • 25
  • 5
0
votes
1 answer

KafkaJSBrokerNotFound: Broker -1 not found in the cached metadata

trying to create topic with kafkajs in node js. using docker containers for Kafka and Zookeeper. getting the above error. what could be the reason? unfortunetly, googling it didn't help. i'm using windows 10 so working with WSL. Thanks const {Kafka}…
Shahar
  • 3
  • 3
0
votes
2 answers

Kafka connection issue when broker is in different Kubernetes namespace

I have a Node.js microservice connecting to a Kafka service. If the Kafka broker is in the default namespace with the other microservices, there are no issues. const kafka = new Kafka({ clientId: 'my-app', // brokers:…
BPDev
  • 397
  • 1
  • 9
0
votes
1 answer

keep kafkajs consumer running when all the associated brokers are down

I have three brokers running on 9002, 9003 and 9004 port . Now i am using kafkajs npm library to create a consumer. Now in the ideal scenario when all the brokers are down at that time i am getting below error…
lal rishav
  • 549
  • 3
  • 5
  • 17
0
votes
1 answer

How can I selectively consume messages from a topic in kafkajs?

Let's say I have the offset and partition of specific messages in a kafkajs topic. How can I attach a consumer to this topic and consume only these specific messages? For example, if I have the offset and partition of a message in a topic, I would…
William Jiang
  • 57
  • 1
  • 4
1 2 3
…
13 14