Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
6
votes
5 answers

Kafka.JS refuses to connect <<[BrokerPool] Failed to connect to seed broker, trying another broker from the list>>

Consider the Kafka Producer : const { Kafka, logLevel } = require('kafkajs'); const kafka = new Kafka({ clientId: 'my-app', brokers: ['kafka1:9092', 'kafka2:9092'], logLevel: logLevel.ERROR, }); const run = async () => { const producer =…
JAN
  • 21,236
  • 66
  • 181
  • 318
6
votes
1 answer

ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread

i am trying to create producer and send some data to it from intellij idea but while running this program i got ERROR like ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-producer-network-thread i have written…
Vishal Barvaliya
  • 197
  • 1
  • 3
  • 14
6
votes
3 answers

Unable to send messages to dead letter topic from consumer in Kafka

'I am trying to route a message to a Dead letter topic in Kafka in case of any failures in processing the corresponding message. I have setup the SeektoCurrentErrorHandler and DeadLetterPublishingRecoverer for this functionality. The consumer throws…
Abhijith Gururaj
  • 437
  • 4
  • 14
6
votes
0 answers

Kafka commitTransaction acknowledgement failure

According to Kafka's commitTransaction documentation, commitTransaction will fail with TimeoutException if it doesn't receive a response in certain time Note that this method will raise TimeoutException if the transaction cannot be committed before…
NinjaTurtle
  • 93
  • 1
  • 6
6
votes
5 answers

Error while fetching metadata with correlation id 92 : {myTest=UNKNOWN_TOPIC_OR_PARTITION}

I have created a sample application to check my producer's code. My application runs fine when I'm sending data without a partitioning key. But, on specifying a key for data partitioning I'm getting the error: [kafka-producer-network-thread |…
Anonymous
  • 151
  • 1
  • 1
  • 6
6
votes
2 answers

Difference between kafka idempotent and transactional producer setup?

When setting up a kafka producer to use idempotent behaviour, and transactional behaviour: I understand that for idempotency we set: enable.idempotence=true and that by changing this one flag on our producer, we are guaranteed exactly-once event…
Ryu S.
  • 1,538
  • 2
  • 22
  • 41
6
votes
2 answers

What happens if offset specified by kafka consumer is not present in Broker?

I am new to kafka. I want to understand how kafka consumer behaves when offset specified by it is not present. Maybe the message at the given offset is deleted due to retention policy or consumer specified an invalid offset value.
Rahul Vedpathak
  • 1,346
  • 3
  • 16
  • 30
6
votes
4 answers

Kafka consumer not consuming from beginning

I have Kafka setup on my local machine and have started the zookeeper and a single broker server. Now i have a single topic with following description: ~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-topics.sh --zookeeper 127.0.0.1:2181…
KayV
  • 12,987
  • 11
  • 98
  • 148
6
votes
1 answer

No exception is coming while sending message when kafka is down

I have scenario where kafka is getting down randomly. When i send message to kafka and it is down i want to handle it separately. But When i send message to kafka while it is down, no exception is coming and so i am not able to identify whether…
Deepank Porwal
  • 73
  • 1
  • 8
6
votes
3 answers

Kafka broker is not available from localhost

I have installed kafka_2.11-1.1.0 and set advertised listener to advertised.listeners=PLAINTEXT://:9092 (in $KAFKA_HOME/config/server.properties). I can connect and write to my kafka using java code and see my cluster via kafka-tool from…
Vahid F
  • 377
  • 1
  • 7
  • 21
6
votes
1 answer

When to close a producer or consumer

Lately, we are having some performance issues with our Kafka consumers and producers. We use Kafka Java API in scala. What is considered to be good practice wrt opening and closing of consumer and producer objects? I believe this is a quite…
Achilleus
  • 1,824
  • 3
  • 20
  • 40
6
votes
3 answers

push multiple Line Text as one message in a kafka topic

I want to push a text consisting of multiple lines as one message into a kafka topic. After I enter: kafka-console-producer --broker-list localhost:9092 --topic myTopic and copy my text: My Text consists of: two lines instead of one I get two…
ScalaNewbie
  • 173
  • 3
  • 12
6
votes
0 answers

How to setup kafka transactional producer

I'm trying to setup kafka transactional producer on windows 10. I downloaded and ran kafka as single node from CLI. Run as follow: .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties .\bin\windows\kafka-server-start.bat…
Andrii Ofim
  • 151
  • 2
  • 4
6
votes
1 answer

Expiring 1 record(s) for xxxxx: 30030 ms has passed since batch creation plus linger time

My use case: Using Postman, I call a Spring boot soap endpoint. The endpoint creates a KafkaProducer and send a message to a specific topic. I also have a TaskScheduler to consume the topic. The problem: When calling soap to push a message to a…
Erick Audet
  • 349
  • 5
  • 13
6
votes
1 answer

kafka producers are very slow

I am new in Kafka and I have a question that I'm not able to resolve. I have installed Kafka and Zookeeper in my own computer in Windows (not in Linux) and I have created a broker with a topic with several partitions (playing between 6 and 12…
Enrique López Moreno
  • 63
  • 1
  • 1
  • 5
1 2 3
…
99 100