Stack Overflow
Stack Overflow

Questions tagged [kafkajs]

KafkaJS is an open-source Node.js client library for Apache Kafka

KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.

199 questions
1
vote
0 answers

Trying to publish a message onto a topic on Kafka from nestjs

I'm trying to publish (producer) a message onto Kafka. The code for receiving/consuming messages is simple enough, but I was hoping to be able to publish some of those back onto a second topic at some point. When I attempt to do that, I get the…
John O
  • 4,863
  • 8
  • 45
  • 78
1
vote
1 answer

NestJS: Global exception filter not catching anything thrown from a Kafka-based microservice app

We're using NestJS as our Typescript framework for a microservice based architecture. Some of our deployments are what we call "Kafka workers", pods that run code that doesn't actually expose any REST endpoints, rather only listens to kafka topics…
Maor Barazani
  • 680
  • 3
  • 12
  • 31
1
vote
1 answer

How to execute the Kafka Consumer file from Javascript

I have used KafkaJS library and configured an consumer code to fetch messages from a topic of one of my streaming API. I have created a consumer.js file and I'm able to run the file from the location in the commandline using node consumer.js But my…
Bijo Joseph
  • 31
  • 1
1
vote
1 answer

Kafka consumer subscribers -- member assignments being lost (nodejs)

Synopsis We have a subscriber that responds to events on 5 topics with a single consumer-group, each with a single partition. When firing events into these topics, we see our code handling the first two events, then nothing. When we check, on…
Software Engineer
  • 15,457
  • 7
  • 74
  • 102
1
vote
0 answers

Unable to setup request/response based messaging in Kafka NestJS microservice

I am trying to create microservice architecture where I have api-gateway and register-service microservice.Every request is handeled by api-gateway after getting user registered I want to send response back.I have set everything up but in…
Digvijay
  • 2,887
  • 3
  • 36
  • 86
1
vote
2 answers

How to commit individual messages on success with KafkaJS

I am trying to find a similar solution to what node-rdkafka does for committing individual messages on success. In node-rdkafka I was able to do call consumer.commit(message); after message processing succeeded. What is the equivalent in KafkaJS? I…
HIT_girl
  • 785
  • 7
  • 23
1
vote
1 answer

Use kafkaJS admin client trough NestJS microservices module

as the title suggest. Is there a possibility to use the admin client functions from KafkaJS trough the nestJS module? I'am trying to have a more detailed grip on the creation of topics trough the API.
Esteday
  • 41
  • 3
1
vote
1 answer

KafkaJs - running consumers in background

I'm writing a service which implements 2 consumers using Node and kafkajs https://kafka.js.org/docs/consuming I have 2 async function which have initialized consumers and in theory are pending to read messages. Something like this: const { Kafka } =…
David Faizulaev
  • 4,651
  • 21
  • 74
  • 124
1
vote
1 answer

How to do unit testing to @Inject with ClientKafka in NestJS

I need to do some unit test to a kafka implementation in my project with NestJS but I don't know how to do it. I have a Service thats inject a Client Kafka export class Service { private static readonly logger = new Logger(ProducerService.name); …
napb
  • 11
  • 2
1
vote
2 answers

Pod publishing messages with KafkaJs to Strimzi keeps crashing

This is my KafkaJs based publisher client. I created a container image and submitted a Pod YAML to Strimzi broker. const { Kafka } = require('kafkajs') async function clients() { const kafka = new Kafka({ clientId: 'my-app', brokers:…
cogitoergosum
  • 2,309
  • 4
  • 38
  • 62
1
vote
1 answer

Is there a way for the consumer the get the n last messages with KafkaJS?

im new to Kafka and KafkaJS. Im trying to find a way to get the last n messages sent to a topic, instead of getting all non-consumed messages of that topic. For example, I have a producer that produces 1 message per second and let it wait for 5…
MrFrenzoid
  • 1,208
  • 6
  • 21
1
vote
1 answer

Question on designing a Kafka Consumer in node.js and making it idempotent

We are trying to write a consumer that would subscribe to a particular topic and do some processing when a couple conditions in the data are met. One operation in the processing however can be done only once, and for the purpose of simplicity let us…
Souvik Majumdar
  • 87
  • 7
1
vote
3 answers

Create AWS amazon-MSK JavaScript Client connection

Has anybody successfully established client connection to Amazon MSK Kafka cluster using JavaScript? No YouTube video or online example AFAIK is out there. Attempts to use KafkaJs npm module are not working for me, because the SASL AWS I am roles is…
Fidelis Chimombe
  • 11
  • 1
  • 2
1
vote
1 answer

Kafka js quick example errors

I have this very simple code which is supposed just to transmit a simple message from producer to consumer using Kafka in local host which come from the official GitHub const { Kafka } = require('kafkajs') const kafka = new Kafka({ clientId:…
Kevin Heirich
  • 109
  • 2
  • 12
1
vote
1 answer

How to get Assigned Partition for a Kafka Consumer in JS

I'm working with a Kafka Consumer using KAFKAJS. I have a requirement to know all the partitions assigned to a specific consumer. For example, let's say Topic 1 has 5 partitions. And there are 2 consumers with the same clientId. So, one will be…
karanraj singh saini
  • 488
  • 1
  • 4
  • 10
1 2 3
…
13 14