Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
9
votes
2 answers

Can single Kafka producer produce messages to multiple topics and how?

I am just exploring Kafka, currently i am using One producer and One topic to produce messages and it is consumed by one Consumer. very simple. I was reading the Kafka page, the new Producer API is thread-safe and sharing single instance will…
Shankar
  • 8,529
  • 26
  • 90
  • 159
9
votes
1 answer

INFO Closed socket connection for client /127.0.0.1:48452 which had sessionid 0x15698f5ac360001 (org.apache.zookeeper.server.NIOServerCnxn)

I installed fresh zookeeper & kafka both. I started them both. Then when I want to see the list of topics with this command: bin/kafka-topics.sh --list --zookeeper localhost 2181 It gives me the socket connection closed. Here is the screen shot: …
Darpan27
  • 239
  • 1
  • 4
  • 9
9
votes
3 answers

Kafka throws java.nio.channels.ClosedChannelException

When i try to consume messages from the kafka server which is hosted in ec2 with kafka console tool (V 0.9.0.1 , i think this uses old consumer APIs) I get following exception. How can i overcome this? #./kafka-console-consumer.sh --zookeeper…
Ratha
  • 9,434
  • 17
  • 85
  • 163
9
votes
1 answer

Kafka Maven Dependencies

What's the difference between the below two dependencies? Do i really need the first one to make a consumer or producer app? org.apache.kafka kafka_2.9.2 …
nikel
  • 3,402
  • 11
  • 45
  • 71
9
votes
2 answers

Kafka Serializer JSON

I am new to Kafka, Serialization and JSON WHat I want is the producer to send a JSON file via kafka and the consumer to consume and work with the JSON file in its original file form. I was able to get it so JSON is converter to a string and sent…
user3780651
  • 139
  • 1
  • 1
  • 5
8
votes
1 answer

Kafka: Sarama, idempotence and transactional.id

Does Shopify/sarama provide an option similar to transactional.id in JVM API? The library supports idempotence (Config.Producer.Idemponent, similar to enable.idempotence), but I don't understand how to use it without transactional.id. Please,…
Timurib
  • 2,735
  • 16
  • 29
8
votes
1 answer

java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.ConsumerGroupMetadata

I am trying to create a simple Kafka producer. I followed a tutorial since I am new to this topic. I created a config file as recommended in the video. Here is the config file I am using. import…
Rohi
  • 385
  • 2
  • 3
  • 15
8
votes
2 answers

Can talk to Zookeeper but not to the message brokers

I'm using kafka-python to produce messages for a Kafka 2.2.1 cluster (a managed cluster instance from AWS's MSK service). I'm able to retrieve the bootstrap servers and establish a network connection to them, but no message ever gets through.…
Kilian Foth
  • 13,904
  • 5
  • 39
  • 57
8
votes
3 answers

Can a message loss occur in Kafka even if producer gets acknowledgement for it?

Kafka doc says: Kafka relies heavily on the filesystem for storing and caching messages. A modern operating system provides read-ahead and write-behind techniques that prefetch data in large block multiples and group smaller logical writes into…
Mahesha999
  • 22,693
  • 29
  • 116
  • 189
8
votes
2 answers

Transactional Producer vs Just Idempotent Producer Java (Exception OutOfOrderSequenceException)

I use spring-kafka with idempotent producer configuration: these are my configuration props: Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, Joiner.on(",").join(appProps.getBrokers())); …
Rakesh Subramanian S
  • 81
  • 1
  • 3
8
votes
4 answers

Guidelines to handle Timeout exception for Kafka Producer?

I often get Timeout exceptions due to various reasons in my Kafka producer. I am using all the default values for producer config currently. I have seen following Timeout exceptions: org.apache.kafka.common.errors.TimeoutException: Failed to…
xabhi
  • 798
  • 1
  • 13
  • 30
8
votes
1 answer

Kafka Producer TimeOutException

I am running a Samza stream job that is writing data to Kafka topic. Kafka is running a 3 node cluster. Samza job is deployed on yarn. We are seeing lot of these exceptions in container logs : INFO [2018-10-16 11:14:19,410]…
Anuj jain
  • 493
  • 1
  • 8
  • 26
8
votes
2 answers

Get Kafka compressed message size

I would like to know the compressed size of a message in kafka. I use kafka 1.1.0 and java kafka-connect 1.1.0 to send messages from my producer to a topic. If the message is too large for my producer, I get a The message is xxx bytes when…
Marius
  • 365
  • 4
  • 18
8
votes
3 answers

What do we mean by 'commit' data in Kafka broker?

In a Kafka cluster containing N brokers , for Topic T against a partition, producers publish data to Leader broker. By the term 'commit' in Kafka terminology , does it mean the data is committed in Leader broker or the data is committed to the…
D Developer
  • 143
  • 1
  • 10
8
votes
1 answer

How to get back Kafka producer and consumer configuration (Java API)?

The use case is following. I am passing producer or consumer reference over many objects instances in Java code. At some of them I would like to do some checks for the Kafka configuration. It means I would like to get back, what effective…
Seweryn Habdank-Wojewódzki
  • 2,037
  • 4
  • 30
  • 51
1 2 3
…
99 100