Stack Overflow
Stack Overflow

Questions tagged [kafkajs]

KafkaJS is an open-source Node.js client library for Apache Kafka

KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.

199 questions
0
votes
1 answer

service not consume message from all topic consumer group on AWS Kafka Cluster

I am very new to AWS and Kafka Cluster. We have NodeJs service which connects to two different Kafka cluster. We have multiple instances of the consumer service and each instance consume a record from every topic but from last couple of weeks we…
ppb
  • 2,299
  • 4
  • 43
  • 75
0
votes
1 answer

KafkaJS producer is disconnected only for transactions

I have a simple app in nodejs where I'm trying to create a transaction in KafkaJS. I've followed the docs and have my client connected and a transactional producer created with a transaction id. The problem is that when I try to run the transaction…
user
  • 1,022
  • 2
  • 8
  • 30
0
votes
1 answer

How to resolve nodejs deprecation warning - RFC 6066

I am trying to create a sample nodejs app for kafkajs producer and consumer. When I start the app, I get the deprecation error. So, I tried with --trace-deprecation flag and get this stack trace. How can I fix it? I tried using different…
adbdkb
  • 1,897
  • 6
  • 37
  • 66
0
votes
0 answers

I am trying to consume kafka message from kafkajs, but not able to read the message,

I am trying to consume kafka message from kafkajs, but not able to read the message, please can you let me know the issue. Any Idea how to resolve this? const kafka = new Kafka({ logLevel: logLevel.INFO, ssl: true, brokers:…
Nandish Kumar
  • 1
  • 2
0
votes
1 answer

Strimzi Kafka ssl validation failed VS AWS MSK ssl passed

I have nodejs app using kafkajs package for connecting to AWS MSK. We are moving to Strimzi Kafka because we already have a kubernetes cluster and we don't need the MSK anymore. Until now we were connected with SSL but didn't have to specify any CA…
yershalom
  • 786
  • 1
  • 8
  • 19
0
votes
0 answers

How to increase batch size of kafka consumer batch

I am using kafka consumer eachBatch: async ({ }) to fetch messages from kafka because our system has to process millions of messages per day and sync the same to Big Query How can I increase batch size to 100 messages?
Rayappan A
  • 1
  • 1
0
votes
1 answer

How to build a spring-kafka consumer to consume messages produced by KafkaJS producer

I know in KafkaJS we can consume message with the following code const run = async () => { await kafkaClient.consumer.subscribe({ topic: 'mytopic', fromBeginning: true }) await kafkaClient.consumer.run({ eachBatchAutoResolve: false, …
John Smith
  • 1
  • 2
0
votes
1 answer

the relationship between Kafka message format and different frames

I'm new to Kafka. I'm a little confused about the kafka message format. I tested a KafkaJS consumer. const run = async () => { await kafkaClient.consumer.subscribe({ topic: 'mytopic', fromBeginning: true }) await kafkaClient.consumer.run({ …
John Smith
  • 1
  • 2
0
votes
1 answer

how to add delay between messages in kafka js consumer

I am listening from a consumer in kafka js and messages are coming too fast, i want to add delay like, 10 messages per 5 seconds how can I achieve that without losing any messages in queue.
mohit
  • 91
  • 6
0
votes
1 answer

How to set up NestJS microservices integration/e2e testing KafkaJS consumer (MessagePattern/EventPattern handler)

I'm proofing out an integration test with NestJS/KafkaJS. I have everything implemented except the function on the event listener (consumer) for the topic I'm emitting to is not being called. I read somewhere you can't consume a message until the…
BrokenWings
  • 169
  • 12
0
votes
0 answers

Node.js WebSocket Server: How to handle individual subscriptions from clients

I want to build a WebSocket server (node.js, express, ws) that can handle individual subscriptions for each client. The ideas is that the client (socket.io) sends a message with a subscription request to the server and the server then sends back the…
CarloP
  • 99
  • 1
  • 12
0
votes
1 answer

Connecting two NestJS gRPC microservices on localhost

Introduction I am quite new to NestJS, but really like how powerful it is so far, so I wanted to see how far I could get with using Kafka, gRPC and NestJS together. My end goal is to have the following design, since I want to replicate my Kafka…
Mattia Bradascio
  • 21
  • 4
0
votes
0 answers

Does Nodejs Kafkajs producer have internal buffer?

This may imply Kafkajs producer doesn't have internal buffer. What would be the best practices to handle heavy traffic(e.g 1k number of messages or 100Mb size per second ) sent out by producer? For example, assume the upper limit (e.g. max num of…
androidkc
  • 689
  • 2
  • 7
  • 16
0
votes
1 answer

Kafkajs - multiple consumers reading from same topic and partition

I'm planning to use Kafkajs https://kafka.js.org/ and implement it in a NodeJs server. I would like to know what is the expected behavior in case I have 2 (or more) instances of the server running, each of them having a consumer which is configured…
David Faizulaev
  • 4,651
  • 21
  • 74
  • 124
0
votes
1 answer

NestJs/microservices Kafka - Consume messages per batch

I'm looking for consuming messages per batch with nestjs/microservices by using the decorator @MessagePattern('topic') or something equivalent. I would like to fetch multiple records together, process them and then, send the commit of the latest…
Ruyik
  • 5
  • 3
1 2 3
…
13 14