Stack Overflow
Stack Overflow

Questions tagged [kafkajs]

KafkaJS is an open-source Node.js client library for Apache Kafka

KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.

199 questions
1
vote
0 answers

How to subscribe to a Kafka topic using NestJS KafkaClient?

I am implementing kafka adapter using NestJS adapters. Within this adapter, I need to be able to publish and subscribe to Kafka topics. The adapter class is not a controller, so I can't add @EventPattern() methods there. I can inject kafkaClient and…
Vololodymyr
  • 1,996
  • 5
  • 26
  • 45
1
vote
1 answer

Kafka js container cannot connect to kafka container on the same docker stack

I have a ELKK docker compose stack and I'm trying to connect a kafka client (kafka js) to kafka but with no luck. docker-compose.yml version: "3.7" services: zookeeper: image: confluentinc/cp-zookeeper:5.4.1 container_name: zookeeper …
Morgy
  • 11
  • 1
  • 2
1
vote
1 answer

Nest.js Inject kafka client to the service

I have a Nest.js microservice application. For message broker between MS I'm using kafka. Now I want to inject this same connection inside health service to test it's state. I don't want to create a new client and new connection. I need the one…
Nika Andriadze
  • 21
  • 4
1
vote
2 answers

How to wait for Kafka respone in a Jest test with KafkaJS?

Consider this test, in which a message is sent from the test to topic 'out', and the tested code is expected to consume it and reply by sending a message to topic 'in'. In order to pass, I want to make sure a message was sent to topic…
chikchakchok
  • 185
  • 1
  • 8
1
vote
1 answer

How to scale kafka consumers in Node.js on Kubernetes

Have a few design questions, would appreciate feedback based on your experiences with Kafka and KafkaJS(any such library): Is partition a way to scale in Kafka? If I create 3 partitions and only 1 consumer, do I loose messages with those 2 non…
maopuppets
  • 420
  • 3
  • 9
  • 27
1
vote
1 answer

Kafka Protobuf Console Consumer Serialization Exception

I have a problem regarding the #protobuf #serialization which occurs in #nodejs and #apache #kafka run by #confluent platform all in one community. I serialize the data with google protobuf or protobufjs and send it to kafka with kafkajs. However,…
Hossein Mayboudi
  • 395
  • 1
  • 4
  • 12
1
vote
1 answer

Is it OK to create topics that already exist in KafkaJS?

I am dynamically creating a series of topics using KafkaJS, and found that in order to do so without creating election problems I should use the createTopics admin feature. The issue is that if createTopics is called on a topic that already exist,…
slifty
  • 13,062
  • 13
  • 71
  • 109
1
vote
1 answer

KafkaJS - Find offset most efficient way

(Complete beginner when it comes to kafka and kafkajs so i'm sorry if this is silly question) I have a problem where we have a topic that retains 48hrs of data (millions of records); And i'm wondering the best way to get from this topic the last "20…
Shanon Jackson
  • 5,873
  • 1
  • 19
  • 39
1
vote
0 answers

Kafka connection error when using KafkaJS

I have installed Kafka on my local dev machine based on the instructions here: https://kafka.apache.org/documentation/#quickstart I am able to produce and consume messages when running the .sh files as explained in the above link. However, when…
Furqan Shaikh
  • 371
  • 2
  • 4
  • 13
1
vote
1 answer

KafkaJS producer ssl certificates

I'm writing a NodeJS Kafka producer with KafkaJS and having trouble understanding how to get the required SSL certificates in order to connect to the Kafka using the SASL-SSL connection. In the KafkaJS documentation there is this configuration for…
Mugetsu
  • 1,739
  • 2
  • 20
  • 41
1
vote
2 answers

OpenShift - Unable to connect Strimzi from KafkaJS

I have Strimzi Kafka cluster set-up successfully on OpenShift. I can see the following services: kafka-brokers kafka-bootstrap zookeeper-client zookeeper-nodes This is actually different from what is called out here; so, not sure, if this is a…
cogitoergosum
  • 2,309
  • 4
  • 38
  • 62
0
votes
1 answer

nodejs application kafka communication over the http_proxy https_proxy proxy getting error

{"level":"ERROR","timestamp":"2023-08-31T15:17:48.211Z","logger":"kafkajs","message":"[Connection] Connection…
Hareesh Chatarasupalli
  • 1
0
votes
0 answers

The broker does not support the requested SASL mechanism

I have an MSK cluster in which i have used AWS IAM authentication .When i am telnet to the broker:9098, the broker gets connected .But when am passing the same broker:9098 in a node app's kafka producer code, it shows "The broker does not support…
ashique
  • 935
  • 2
  • 8
  • 26
0
votes
1 answer

:"kafkajs","message":"[Connection] Connection timeout","broker":"b-2.ashiqxxxxxxxonaws.com:9092

I am using IAM Authentication in my AWS MSK Cluster. but when I am trying to access the MSK broker using telnet , telnet b-1.ashxxxx.xx74.cx.kafka.xx-southeast-1.amazonaws.com 9092, its getting timeout error."kafkajs","message":"[Connection]…
ashique
  • 935
  • 2
  • 8
  • 26
0
votes
1 answer

Error listing topics: KafkaJSConnectionClosedError: Closed connection,Error starting consumer:::: KafkaJSConnectionClosedError: Closed connection

I have an AWS MSK cluster that is using IAM authentication.I am trying to connect with my node app, but I get an error in the consumer part of Kafka. my producer code is : const { Kafka,logLevel } = require('kafkajs'); const kafka = new Kafka({ …
ashique
  • 935
  • 2
  • 8
  • 26
1 2 3
…
13 14