Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
10
votes
1 answer

Ordering guarantees when using idempotent Kafka Producer

I am using Kafka 1.0.1 in my application and I have started using the Idempotent Producer feature that was introduced in 0.11, and I've having trouble understanding the ordering guarantees when using the Idempontent feature. My producer's…
Orr Ganani
  • 101
  • 1
  • 3
10
votes
1 answer

What happens if I don't close the kafka producer

I'm processing xml's and I need to send a message per record, when I receive the last record I close the kafka producer, the problem here is that the send method of the kafka producer is async, therefore, sometimes when I close the producer it trows…
Alejandro Agapito Bautista
  • 1,344
  • 1
  • 15
  • 35
10
votes
2 answers

Difference between request.timeout.ms and timeout.ms properties of Kafka producer

Refering the kafka documentation, a Kafka message producer configuration have request.timeout.ms and timeout.ms properties. Reading the description of these two properties, I am not able clearly distinguish between them. Can anyone explain the…
vatsal mevada
  • 5,148
  • 7
  • 39
  • 68
10
votes
1 answer

Kafka Offset after retention period

I have a kafka topic with 1 partition. if it had 100 messages in it the offset would be from 0.99. According the kafka retention policy all of the messages will be wiped out after the specified period. and i am sending 100 new messages to the topic…
AKC
  • 953
  • 4
  • 17
  • 46
10
votes
4 answers

WARN Error while fetching metadata with correlation id 1 : {MY_TOPIC?=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient)

When I run the following command with kafka 0.9.0.1, I get these warnings[1]. Can you please tell me what is wrong with my topics? (I'm talking to the kafka broker which runs in ec2) ./kafka-console-consumer.sh --new-consumer --bootstrap-server…
Ratha
  • 9,434
  • 17
  • 85
  • 163
9
votes
2 answers

Kafka producer throws an error Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION

My kafka producer is throwing an error "Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION". Here is what I am trying to achieve - KafkaProducer producer = new…
Vishal Pawar
  • 217
  • 4
  • 14
9
votes
3 answers

Kafka design questions - Kafka Connect vs. own consumer/producer

I need to understand when to use Kafka connect vs. own consumer/producer written by developer. We are getting Confluent Platform. Also to achieve fault tolerant design do we have to run the consumer/producer code ( jar file) from all the brokers ?
Anirban
  • 257
  • 4
  • 12
9
votes
2 answers

Apache Kafka the order of messages in partition guarantee

Read this article about message ordering in topic partition: https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2 Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially…
bausov
  • 387
  • 3
  • 12
9
votes
2 answers

How to fix kafka.common.errors.TimeoutException: Expiring 1 record(s) xxx ms has passed since batch creation plus linger time

I am using kafka_2.11-2.1.1 and Producer using spring 2.1.0.RELEASE. I am using spring while I am sending the messages to Kafka topic my producer generates a lot of TimeoutExceptions org.apache.kafka.common.errors.TimeoutException: Expiring 1…
BdEngineer
  • 2,929
  • 4
  • 49
  • 85
9
votes
2 answers

How does kafka decides the partition if I don't mention any

This is how I produce message : String json = gson.toJson(msg); ProducerRecord record = new ProducerRecord<>(kafkaProducerConfig.getTopic(), json); long startTime = System.currentTimeMillis(); try { RecordMetadata meta =…
Rakesh Pullayikodi
  • 101
  • 1
  • 9
9
votes
3 answers

How to create topic in Kafka with Python-kafka admin client?

is there any Python kafka admin client avilable to create topic/delete topic from python program ? I found some python apis but none of them have Admin api available ? Does confluent have python admin api ?
Bharat
  • 311
  • 3
  • 11
9
votes
1 answer

Kafka streams.allMetadata() method returns empty list

So I am trying to get interactive queries working with Kafka streams. I have Zookeeper and Kafka running locally (on windows). Where I use the C:\temp as the storage folder, for both Zookeeper and Kafka. I have setup the topic like…
sacha barber
  • 2,214
  • 1
  • 24
  • 37
9
votes
6 answers

How to send messages synchronously in kafka?

One way of achieving it could be by setting the properties parameter max.in.flight.requests.per.connection = 1. But I want to know if there is an even direct or alternate way of sending messages synchronously in kafka, something like…
user8339674
9
votes
3 answers

org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value

I'm getting this error when running my producer class in Eclipse: org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value Here is my producer class: public class SimpleProducer…
Amritesh Kumar
  • 207
  • 1
  • 5
  • 16
9
votes
4 answers

How to create Custom serializer in kafka?

There is only few serializer available like, org.apache.kafka.common.serialization.StringSerializer How can we create our own custom serializer ?
Vimal Dhaduk
  • 994
  • 2
  • 18
  • 43
1 2 3
…
99 100