Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
30
votes
1 answer

Kafka set compression type at producer vs topic

What's the difference between the following ways of enabling compression in kafka: Approach 1: Create a topic using the command: bin/kafka-topics.sh --create --zookeeper localhost:2181 --config compression.type=gzip --topic test Approach 2: Set the…
shants
  • 612
  • 1
  • 7
  • 12
25
votes
2 answers

Is a web frontend producing directly to a Kafka broker a viable idea?

I have just started learning Kafka. So trying to build a social media web application. I am fairly clear on how to use Kafka for my backend ( communicating from backend to databases and other services). However, I am not sure how should frontend…
user3271166
  • 573
  • 1
  • 6
  • 17
25
votes
6 answers

How to configure logging for Kafka producers?

I am using Kafka producer client and i don't have any log4j configuration in my project. On running, the program prints a lot of Kafka Debug logs which i really don't want. So, i tried to add a log4j.properties to set log level to ERROR as below…
nikel
  • 3,402
  • 11
  • 45
  • 71
24
votes
8 answers

kafka failed authentication due to: SSL handshake failed

I have to add encryption and authentication with SSL in kafka. This is what I have done: Generate certificate for each broker kafka: keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey Create CA. The generated CA is a…
xRobot
  • 25,579
  • 69
  • 184
  • 304
22
votes
1 answer

Reproducing UnknownTopicOrPartitionException: This server does not host this topic-partition

We have encountered few exception on production environment: UnknownTopicOrPartitionException: This server does not host this topic-partition As per my analysis, one possible workaround for this issue is increasing no of retries since this is a…
vatsal mevada
  • 5,148
  • 7
  • 39
  • 68
21
votes
4 answers

Kafka Connect - Failed to flush, timed out while waiting for producer to flush outstanding messages

I am trying to use the Kafka Connect JDBC Source Connector with following properties in BULK…
David
  • 251
  • 1
  • 2
  • 5
20
votes
1 answer

Kafka producer difference between flush and poll

We have a Kafka consumer which will read messages and do so stuff and again publish to Kafka topic using below script producer config : { "bootstrap.servers": "localhost:9092" } I haven't configured any other configuration like…
shakeel
  • 801
  • 1
  • 8
  • 24
20
votes
1 answer

How to send message to a particular partition in Kafka?

I have created a topic that has many partitions. Using the console producer I want to send messages to particular partitions and view the through the console consumer. On the console producer I have tried this, kafka-console-producer.bat…
Amrutha Jaya Raj
  • 612
  • 2
  • 10
  • 32
20
votes
3 answers

Understanding the max.inflight property of kafka producer

I work on a bench of my Kafka cluster in version 1.0.0-cp1. In part of my bench who focus on the max throughput possible with ordering guarantee and no data loss (a topic with only one partition), need I to set the…
Quentin Geff
  • 819
  • 1
  • 6
  • 21
20
votes
6 answers

Error to serialize message when sending to kafka topic

i need to test a message, which contains headers, so i need to use MessageBuilder, but I can not serialize. I tried adding the serialization settings on the producer props but it did not work. Can someone help me? this…
Tiago Costa
  • 1,004
  • 4
  • 17
  • 31
19
votes
2 answers

Using Kafka Producer by different threads

I have kafka producer for my java based web application to push messages to Kafka. As per the documentation I could see kafka producer is thread safe. Does it mean that I can have single instance of Kafka producer and use it by different threads (…
Bill Goldberg
  • 1,699
  • 5
  • 26
  • 50
18
votes
3 answers

Apache-Kafka, batch.size vs buffer.memory

I'm trying to figure out the difference between the settings batch.size and buffer.memory in Kafka Producer. As I understand batch.size: It's the max size of the batch that can be sent. The documentation describes buffer.memory as: the bytes of…
calleman123
  • 181
  • 1
  • 1
  • 3
17
votes
1 answer

Apache Kafka - linger.ms and batch.size settings

In kafka producer settings, what is the expected behavior if you have linger.ms set to 0 and a non zero batch.size? How long will the producer wait for batch.size to be reached before sending the messages? Will it wait forever till the size of…
Vivek Agarwal
  • 301
  • 1
  • 2
  • 8
17
votes
5 answers

Kafka with .Net Client

I want to use kafka on our production environment. I want to know if the latest version of client is bug free for production launch. Is it working with consumer group? I want to pass 10000 records per second, is it suitable for it or not?
vicky garg
  • 181
  • 1
  • 1
  • 4
16
votes
4 answers

Can a Kafka producer create topics and partitions?

currently I am evaluating different Messaging Systems. There is a question related to Apache Kafka which I could not answer myself. Is it possible for a Kafka producer to create topics and partitions (on existing topics aswell) dynamically? If yes,…
smartwepa
  • 311
  • 1
  • 3
  • 11
1
2
3
…
99 100