Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
5
votes
0 answers

How to handle connection issues with kafka using the python kafka library?

I have a serverless function that's trying to send some data to kafka. Sometimes it works and sometimes the connection just drop and the data is lost. The reason for this is that the library for kafka is not raising the exception but it's adding…
Yuv Joodhisty
  • 71
  • 1
  • 4
5
votes
3 answers

Kafka Producer publishing message to single partition

I am new to Kafka and going through the official documentation that is available. On my local system i have started a single kafka instance alongwith the zookeeper. Zookeper and kafka server both are running on default ports. I have created a topic…
ANKIT SRIVASTAVA
  • 133
  • 4
  • 14
5
votes
1 answer

How to identify which messages successfully published to kafka topic and those which failed

Publishing a list of messages to apache kafka. Could anyone provide sample code using the kafka api showing how to identify which messages got successfully published to the topic and those which failed from the response? (Note I'm sending the list…
YheCoder
  • 61
  • 1
  • 7
5
votes
0 answers

kafka producer metrics not exposed on http endpoint with Actuator

I'm using Spring boot 2, Kafka 2.1.4 and I want to expose the kafka Producer metrics in the actuator/metrics endpoint of spring boot actuator pom.xml org.springframework.boot …
Ez-zaidi
  • 86
  • 1
  • 8
5
votes
2 answers

kafka asynchronous send not really asynchronous?

I am using KafkaProducer from the kafka-client 1.0.0 library, and as per the documentation, the method Future send(ProducerRecord record) will immediately return but actually, but looks like not. This method also calls another…
Azel
  • 345
  • 4
  • 15
5
votes
1 answer

Understanding Kafka Message Byte Size

How do I get the size of a single record in Kafka? There's some exposition as to why I need this. This does not appear to be the serializedValueSize exposed on the ConsumerRecord or RecordMetadata classes. I don't really understand the value of this…
xerige2
  • 101
  • 3
  • 9
5
votes
2 answers

Consumer and producer failing with error: "Connection to 0 was disconnected before the response was read"

I have a cluster of 3 Kafka brokers with replication factor of 3 for all topics. Since last few days I am facing this issue, suddenly(few times in a day) consumers and producers are stuck while getting response even when Kafka is running on all 3…
Bhanu Pasrija
  • 137
  • 2
  • 9
5
votes
1 answer

Different between KafkaProducer.close() and KafkaProducer.flush()

Looking at the documentation, I'm not sure if I understand the difference between using close() and flush(). This is the doc for flush() * Invoking this method makes all buffered records immediately available to send (even if…
Javier Holguera
  • 1,301
  • 2
  • 11
  • 27
5
votes
1 answer

WSO2 SP - Kafka source with JSON attributes

I'm trying to read JSON data from Kafka, using following code: @source(type = 'kafka', bootstrap.servers = 'localhost:9092', topic.list = 'TestTopic', group.id = 'test', threading.option = 'single.thread', @map(type = 'json')) define stream…
Manu
  • 75
  • 7
5
votes
2 answers

Kafka Streams does not increment offset by 1 when producing to topic

I have implemented a simple Kafka Dead letter record processor. It works perfectly when using records produced from the Console producer. However I find that our Kafka Streams applications do not guarantee that producing records to the sink topics…
DVS
  • 783
  • 7
  • 25
5
votes
2 answers

Is this possible? producer batch.size * max.request.size > broker max.message.bytes

Average message size is small, but size is vary. Average message size: 1KBytes 1MBytes message incomes in arbitrary rate. / So, producer's max.request.size = 1MBytes broker's max.message.bytes = 2MBytes My questions. To avoid producing size…
Youngrok Ko
  • 351
  • 1
  • 4
  • 10
5
votes
2 answers

Unable to create topic when kafkaProducer tries to send record for the first time INVALID_REPLICATION_FACTOR

I am getting following error in the logs when trying to publish first message to a new topic. [WARN ] [o.a.kafka.clients.NetworkClient][[Producer clientId=producer-1] Error while fetching metadata with correlation id 766890 :…
enator
  • 2,431
  • 2
  • 28
  • 46
5
votes
1 answer

Metadata information from kafka

I am new to Confluent/Kafka and I want to find metadata information from kafka I want to know list of producers list of topics schema information for topic Confluent version is 5.0 What are classes (methods) that can give this information? Are…
user1393608
  • 1,299
  • 3
  • 16
  • 29
5
votes
2 answers

Kafka Consumer Error

I am using a kafka produer and a Spring kafka consumer. I am using a Json serializer and deserializer. Whenever I try to read messages in the consumer from the topic i get the following error: org.apache.kafka.common.errors.SerializationException:…
jenjen
  • 163
  • 1
  • 1
  • 10
5
votes
0 answers

Why is this test taking so long?

I have the following xUnit (version 2.3.1) test for sending 100 messages to Kafka [Fact] public void Test1() { Stopwatch sw = new Stopwatch(); sw.Start(); var config = new Dictionary { …
Markus Deibel
  • 1,261
  • 20
  • 26
1 2 3
…
99 100