KafkaJS is an open-source Node.js client library for Apache Kafka
KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.
Questions tagged [kafkajs]
199 questions
0
votes
0 answers
node-rdkafka not working on docker (no ready event)
We are running a node service on docker (image node:lts-bullseye)
We are currently using kafkajs, which is really easy to use, the problem is that sometimes after the Kafka rebalances and moves topics from one broker to another we can produce…
Dor Gross
- 31
- 3
0
votes
1 answer
Why does Kafka JoinGroup request takes a long time when connecting a new consumer?
So, to keep it short:
Procedure:
Frontend -\> start node -\> consume messages -\> send result back to frontend
Once I start a node and the first request is processed, the results get sent back in an instant.
However if I send the second request…
Pixel2203
- 1
- 1
0
votes
1 answer
Stream from MSK to RDS PostgreSQL with MSK Connector
I've been going in circles with this for a few days now. I'm sending data to Kafka using kafkajs. Each time I produce a message, I assign a UUID to the message.key value, and the the message.value is set to an event like this and then…
RusskiT
- 106
- 7
0
votes
0 answers
Decode org.apache.kafka.connect.data.Decimal in typescript
In our Avro scheme we have following 2 decimal types
{
"name": "LONG",
"type": {
"type": "bytes",
"scale": 12,
"precision": 64,
"connect.doc": "Long",
…
Dimi
- 23
- 5
0
votes
1 answer
How can I make my KafkaJS consumer wait indefinitely for a broker to come online?
I have a KafkaJS consumer configured as follows:
// Create the kafka client
const kafka = new Kafka({
clientId,
brokers,
});
// Create the consumer
const consumer = this.kafka.consumer({
groupId,
heartbeatInterval: 3000,
sessionTimeout:…
bapin93
- 162
- 10
0
votes
1 answer
Dynamic Module's service import
I have a global dynamic module with a service named KafkaService, I want to use this service in my AuthenticationService, but NestJS seems can't inject the KafkaService
I have a KafkaModule which is a Dynamic and Global module
import {…
Rahandi Noor Pasha
- 41
- 8
0
votes
1 answer
Kafka consumer is reading a lot more bytes than expected
TLDR: In essence, we produce about 800 kB/s while a consumer is reading 130 MB/s, which is more than 150 x what we produce ...
A litte more detail. We have a few Golang producers that write about 800 kB/s to a few topics in a Redpanda (Kafka)…
NorwegianClassic
- 935
- 8
- 26
0
votes
0 answers
KafkaJS consumer disconnects after connecting to Kafka container
I have implemented a typescript application containing the "Kafka JS" library to initialize a Kafka consumer within it.
This is the code of the consumer:
export class KafkaConsumer {
public kafka: Kafka = {} as Kafka;
public consumer: Consumer =…
Or S
- 3
- 1
0
votes
1 answer
KafkaJS consume from multiple topics with one consumer
I want to implement retry logic while consuming from Kafka topic using KafkaJS, so basically, I will have 2 topics main-topic and retry-topic and I will
read from -> main-topic
if processing fails | -> retry topic
so is it a bad practice to use one…
Kaneki21
- 1,323
- 2
- 4
- 22
0
votes
0 answers
Performance Kafka API Integration
I'm implementing an integration in two softwares. Basically I need to convert records from a database to another database, however, I need to convert the foreign keys, adapting them to the database that will receive it.
To do this i used a debezium…
Lucas Rissi
- 112
- 3
0
votes
0 answers
Which configuration to use to connect to kafka cluster using Kafka.js
I am not being able to connect to Kafka using kafka.js. I am thinking that I am using the wrong kafka.js configuration.
Following are the items that are given to me :-
Host: kafka-xxx-xx.cloudclusters.net
Port: xxxxx
IP: xx.xx.xxx.xxx
Security…
Abdullah Ch
- 1,678
- 1
- 13
- 31
0
votes
0 answers
Can't update global variable inside of run consumer's function kafka js
I have global variable result_crnt. I try to update it inside of consumerRun function:
global.result_crnt = '';
async function consumerRun(){
try{
await consumer.run({
eachMessage: async({ topic, partition, message }) => {
…
Sara
- 41
- 4
0
votes
0 answers
Consumer doesn't get message kafka-js
I have a GET request (using express-js):
api.get("/test/req", async function(req, res){
try{
let result = '';
var reqId = req.id;
await producer.send({
topic: `producer${clusterWorkerId}`,
…
Sara
- 41
- 4
0
votes
0 answers
Can someone tell in kafkaJs how is default value is assigned
I have a component using kafkaJs library, it has default connectionTimeout, I have tried searching where it is set can't find, please help me find it, I know I can set it in client properties, but I want to know from where default value, comes.
Aman Kandari
- 37
- 5
0
votes
1 answer
Producer.send() in Kafkajs logs "ERROR: [Connection] Connection timeout" but still writes to stream
I'm fairly new to working with Kafka and MSK in AWS. I'm using kafkajs to write from a lambda to an MSK cluster. My records are being written successfully to my Kafka cluster, but my client is also logging connection timeout errors into CloudWatch.…
RusskiT
- 106
- 7