KafkaJS is an open-source Node.js client library for Apache Kafka
KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.
Questions tagged [kafkajs]
199 questions
1
vote
1 answer
TypeError: Error creating bean with name 'kafkaHighLevelConsumer': in docker
I am starting Zookeeper, kafka and kafdrop with docker-compose in local, everything is works.
when I want to do the same thing inside EC2 instance I get this error.
the EC2 type that I'm using is t2.micro with an OBS in the default VPC and…
Ayoub Ba-haddou
- 21
- 3
1
vote
1 answer
How multiple Kafka Consumers in the same consumer group read messages from one partition in the topic?
I would like to know about how the consumers in the same consumer group read the messages from one topic which has only one partition.
For example, I have 3 consumers in one consumer group and that group is polling messages from Topic A which has…
K Bariya
- 13
- 2
1
vote
0 answers
KafkaJSConnectionError: Connection timeout. SSL configuration on Kafka.js
The credentials that I am provided with :- (I also have kafka.keystore.jks and kafka.truststore.jks)
host: xxxxx-xxxxx-x.cloudclusters.net
port: xxxxx
ip: xxx.xxx.xxx.xx
trustore pw: xxxxxxxx
keystore pw: xxxxxxxx
import WebSocket from 'ws';
import…
Abdullah Ch
- 1,678
- 1
- 13
- 31
1
vote
1 answer
Infinite retries when using RpcFilter in NestJS microservice setup with Kafka
I am new to Kafka and I am experiencing a mixed behaviour when trying to setup proper error handling on my consumer when there is an error. In few instances I am observing retry policy in action - kafka retries my message 5 times(as what I…
Clara
- 91
- 2
- 3
1
vote
0 answers
NESTJS Kafka, How to start the microservice when broker is not available
I am using the NestJS microservice with Kafka.
Issue: Let's say at server startup the Kafka is not available, the service will crash as the server is not able to connect to the service.
Expected Behaviour: I will like the service to get started even…
JDev
- 1,662
- 4
- 25
- 55
1
vote
1 answer
How to throw custom error for KafkaJS client when initialization fails?
I am using the following code to initialize the Kafka client:
this.kafka = new Kafka({
clientId: ,
brokers: [
`${process.env.KAFKA_BROKER_1}`,
`${process.env.KAFKA_BROKER_2}`,
`${process.env.KAFKA_BROKER_3}`,
],
…
Blueboye
- 1,374
- 4
- 24
- 49
1
vote
0 answers
This server is not the leader for that topic-partition in KafkaJS package
I am using the kafkaJS library to connect Kafka consumer, below is the configuration
private createKafkaConsumer(): Consumer {
const kafka = new Kafka({
logLevel: logLevel.WARN,
brokers: this.kafkaHost,
connectionTimeout:…
Mohamed Anser Ali
- 1,269
- 2
- 8
- 13
1
vote
0 answers
KafkaJS reconnecting after initial connection with kafka and kafka connect, throwing error?
I am trying to connect with kafka with my kafka microservice,
connection configuration kafkajs
but throwing following error :
errors even after initial connection
Ny best guess it is something regarding my kafka environment ,
which is like following…
Ibrahim Sajid
- 11
- 1
1
vote
1 answer
kafkajs run eachMessage is not fetching messages
I'm trying to fetch messages from Kafka using kafkajs
I want to catch all messages into "messages" (an array)
config - the variable which hold the configs
Here is the function I'm using:
const { Kafka, logLevel } = require('kafkajs')
async function…
Gil Allen
- 1,169
- 14
- 24
1
vote
0 answers
NestJS Kafka Microservice - RpcException
I'm very new to Kafka/microservices, and I'm not understanding how error handling should work on these.
If I have no RpcException filter in place, the default RpcExceptionHandler seems to just catch the error and the entire microservice application…
DaleCooper123
- 11
- 1
1
vote
0 answers
Can kafkajs consume messages with different schemas from the same topic using the same consumer?
A topic has messages with different schemas using RecordNameStrategy.
Can a consumer implemented with kafkajs (version 3.0.1) consume in the same batch those messages and parse them?
Actually, from the source code seems it can, since it gets the…
YFl
- 845
- 7
- 22
1
vote
0 answers
kafkajs consumer not working for topic with more than one partition
When i try to consume message from topic with more than one partition, consumer not working at all, this is my sample code for initialize kafka and producer:
const kafka = new Kafka({
clientId: config.kafka.client,
// eslint-disable-next-line…
Meysam Goodarzi
- 11
- 4
1
vote
1 answer
Kafka messages ordering not being respected on the last two messages
I'm working on a NestJS project (hybrid application) and we use KafkaJS to exchange some data between our microservices and in some cases, the order in which those data are sent to other services is very important as it cannot process the second…
Lucas
- 491
- 6
- 21
1
vote
0 answers
Fails to encode Proto payload containing a Timestamp field
I'm using kafkajs and @kafkajs/confluent-schema-registry in Typescript in order to encode a message payload using proto and send it to Kafka. When I try to encode the message, it fails with the following…
alerojlop
- 61
- 6
1
vote
1 answer
how npm overrides works or how to use it
In my nodejs application's package.json i have some dependency
"dependencies"{
"some-v2":"git commit hash path"
}
Inside this dependency I am using kafkajs old version. To override with new version I am using npm overrides feature as below…
ppb
- 2,299
- 4
- 43
- 75