KafkaJS is an open-source Node.js client library for Apache Kafka
KafkaJS is an open-source Node.js client library for Apache Kafka, a distributed streaming platform designed to store and process high-throughput data streams.
Questions tagged [kafkajs]
199 questions
0
votes
1 answer
Detect and control on clientside when a kafkajs consumer or producer timeouts
Im new to kafka and to kafkajs.
I have made a Kafka producer and consumer using kafkaJS.
I want to know how can I control what happens on my js code when the consumer or producer times out on the connection with the kafka server.
I imagine there…
MrFrenzoid
- 1,208
- 6
- 21
0
votes
1 answer
ResponseError: Confluent_Schema_Registry - Error, status 400
I'm using kafkajs and confluent-schema-registry in Nodejs. I can log the message.value (avro) but I'm having an error when trying to decode it. It says: ResponseError: Confluent_Schema_Registry - Error, status 400
const payload = await…
Joy Lab
- 584
- 5
- 6
0
votes
1 answer
KafkaJS: how to run a consumer on a remote website/server?
I am trying to stream live data using kafka to our company's website so that I can display a live updating plot.
I have a working JS consumer which I wrote with help from the kafkaJS docs.
Now I want to call this script from an HTML file on our…
TechnoTerry
- 65
- 2
- 6
0
votes
0 answers
How to test send message in kafka with jest?
export const produce = async (topic: string, url: string): Promise => {
await producer.connect();
await producer.send({
topic,
compression: CompressionTypes.GZIP,
messages: [{ key: `key-${Math.random() * 100000}`,…
Tri Bui Quang
- 61
- 6
0
votes
1 answer
Connection timeout when I try to connect to AWS MSK brokers
I created a cluster with 2 brokers in MSK but I cant connect using kafkajs client.
I keep getting this connection timeout error
{"level":"ERROR","timestamp":"2021-07-29T13:35:35.625Z","logger":"kafkajs","message":"[Connection]
Connection…
Leutrim Osmani
- 1
- 1
- 1
0
votes
1 answer
How to stop receiving kafka-metadata messages on a topic
For some reason, sometimes, our Confluent Kafka consumer receives wierd messages on a specific topic:
{
"magicByte": 2,
"attributes": 0,
"timestamp": "1623227829187",
"offset": "11575814",
"key": null,
"value": {
"type": "Buffer",
…
Stav Alfi
- 13,139
- 23
- 99
- 171
0
votes
1 answer
Why does Promise.race not resolve in kafkajs eachMessage callback
I have defined a promise like this ...
const result = await Promise.race([
new Promise(resolve => {
consumer.run({
eachMessage: ({ message }) => {
const data = JSON.parse(message.value.toString());
…
Jiew Meng
- 84,767
- 185
- 495
- 805
0
votes
1 answer
Using Kafka to ingest CSV data and then have multiple consumers subscribe and create new CSV files which are a subset of the original
I was watching one of Robin Moffatt's videos (https://rmoff.net/2020/06/17/loading-csv-data-into-kafka/) and believe Apache Kafka might help me automate a workflow I have.
I have a requirment where I need to ingest a CSV from a customer, send off a…
Greg
- 147
- 2
- 9
0
votes
1 answer
Error while creating Topics using KafkaJS
I am using the KafkaJS to create the topics.
const kafka = new Kafka({
connectionTimeout: 10_000,
authenticationTimeout: 10_000,
brokers: [`HOST_NAME:9092`],
clientId: 'example-producer',
ssl: {
servername: 'HOST_NAME',
…
user1912
- 35
- 1
- 8
0
votes
1 answer
Read a kafka topic and expose the data via Rest API for prometheus to scrape ( Nodejs)
I have used kafkajs to expose the data read from a kafka topic to be exposed via a http endpoint for prometheus to scrape the data. But I am not able to expose the data from kafka topic. I have written a producer and consumer like this
Producer.js
…
Karthik
- 377
- 3
- 12
0
votes
1 answer
How to have multiple Kafka brokers replication
So I'm new to kafka, and I have trouble finding informations on having multiple kafka instances in order to have my message service still up if a broker instance goes down.
I've made a little local demo with Kafka JS and I've seen that we declare…
Kevin Heirich
- 109
- 2
- 12
0
votes
1 answer
How do I connect Kafkajs with Socket.io
I've been studying kafkajs and socket.io I'm am very new to it and i cant seem to understand some things.
I have created a chat application that basically by opening a browser(client) you can type messages and they get displayed in a chat-window.
I…
Non Sin
- 5
- 1
- 2
0
votes
1 answer
Is there any way for a Consumer to send an acknowledgement to a producer implemented using kafka(kafka.js) and node.js?
I believe the 3 types of acknowledgement as the producer properties are limited to the leader and the producer, I want the producer to receive a concrete message when the consumer consumes a message from the storage/queue through kafka broker.
Also…
rajas_m
- 45
- 9
0
votes
1 answer
kafkajs - Wrong password for SASL authentication did not raise an error
This is my producer using kafkajs. When I provide incorrect user name in options.username, I see the message Connected. How do I handle connection errors with kafkajs?
const { Kafka, logLevel } = require('kafkajs')
async function…
cogitoergosum
- 2,309
- 4
- 38
- 62
0
votes
1 answer
Unexpected token in kafkajs code for a kafka consumer application
Nodejs version : 4.4.7
NPM version: 2.15.8
I am running the following code, which is literally copied from the following URL
https://kafka.js.org/docs/getting-started
const { Kafka } = require('kafkajs')
const kafka = new Kafka({
…
Nick
- 157
- 2
- 14