Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
8
votes
2 answers

Kafka producer send blocks indefinitely when kafka servers are down

I'm using Kafka 0.11.0.0. I have a test program that publishes to a Kafka topic; if the zookeeper and Kafka servers are down (which is normal in my development environment; I bring them up as needed) then the call to KafkaProducer<>.send() hangs…
Will Duquette
  • 116
  • 1
  • 6
8
votes
2 answers

What's the best way to design message key in Kafka?

I have a partitioned topic, which has X partitions. As of now, when producing messages, I create Kafka's ProducerRecord specifying only topic and value. I do not define a key. As far as I understand, my messages gonna be distributed evenly amongst…
Ihor M.
  • 2,728
  • 3
  • 44
  • 70
8
votes
1 answer

How to add consumer group to message in java?

I'm new in java, spring and kafka I have the next code for sending message kafkaTemplate.send(topic, message); My configuration for producer: props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); …
pasha
  • 566
  • 5
  • 19
8
votes
3 answers

Kafka DefaultPartitioner algorithm

There is a very small but very powerful detail in the Kafka org.apache.kafka.clients.producer.internals.DefaultPartitioner implementation that bugs me a lot. It is this line of code: return DefaultPartitioner.toPositive(Utils.murmur2(keyBytes)) %…
nucatus
  • 2,196
  • 2
  • 21
  • 18
8
votes
2 answers

Kafka multiple partition ordering

I am aware that it is not possible to order multiple partitions in Kafka and that partition ordering is only guaranteed for a single consumer within a group (for a single partition). However with Kafka Streams 0.10 is it now possible to achieve…
user983022
  • 979
  • 1
  • 18
  • 30
8
votes
2 answers

How does kafka ack batch AsyncProducer

How does kafka sends ack when using batch async Producer? Is the ack is per message/per batch/per sub-batch(i.e. batch per partition)? Is it recommended to use the ack in async batch prdocuer? or better just to use the callback mechanism?
dna
  • 81
  • 3
8
votes
3 answers

Kafka 0.10 Java Client TimeoutException: Batch containing 1 record(s) expired

I have a single node, multi (3) broker Zookeeper / Kafka setup. I am using the Kafka 0.10 Java client. I wrote following simple remote (on a different Server than Kafka) Producer (in the code I replaced my public IP address with MYIP): Properties…
Armen
  • 81
  • 1
  • 1
  • 5
8
votes
1 answer

apache- kafka with 100 millions of topics

I'm trying to replace rabbit mq with apache-kafka and while planning, I bumped in to several conceptual planning problem. First we are using rabbit mq for per user queue policy meaning each user uses one queue. This suits our need because each user…
Hyounmin Wang
  • 101
  • 2
  • 7
8
votes
4 answers

Error reading field 'topic_metadata' in Kafka

I am trying to connect to my broker on aws with auto.create.topics.enable=true in my server.properties file. But when I am trying to connect to broker using Java client producer I am getting the following error. 1197 [kafka-producer-network-thread…
user2966021
  • 423
  • 1
  • 7
  • 20
8
votes
1 answer

Does Kafka guarantee message ordering within a single partition with ANY config param values?

If I set Kafka config param at Producer as: 1. retries = 3 2. max.in.flight.requests.per.connection = 5 then its likely that Messages within one partition may not be in send_order. Does Kafka takes any extra step to make sure that messages within a…
user3851499
8
votes
2 answers

Guaranteed delivery of multiple messages to Kafka cluster

If I publish several messages in a row to a Kafka cluster (using the new Producer API), I get a Future from the producer for each message. Now, assuming I have configured my producer to have max.in.flight.requests.per.connection = 1 and retries > 0…
Manjabes
  • 1,884
  • 3
  • 17
  • 34
8
votes
3 answers

how to get acknowledgement from Kafka broker if message is produced by producer?

I would like to get some response from the broker when I produce a message. I have tried CallBack mechanism (by implementing CallBack) used in KafkaProducer.send but it did not work and does not call onCompletion method. When I shutdown Kafka…
usman
  • 1,351
  • 5
  • 23
  • 47
8
votes
6 answers

Error in Publishing message to Kafka topic

I am new to Kafka and try to setup enviroment for it . I am trying to run a single node Kafka but I am getting error in that. Following following steps on mac 1. brew install zookeeper 2. brew install kafka 3. zkServer start 4. …
Alok
  • 1,374
  • 3
  • 18
  • 44
7
votes
2 answers

What is the difference between Kafka partitions and Kafka replicas?

I created 3 Kafka brokers setup with broker id's 20,21,22. Then I created this topic: bin/kafka-topics.sh --zookeeper localhost:2181 \ --create --topic zeta --partitions 4 --replication-factor 3 which resulted in: When a producer sends message…
ace
  • 11,526
  • 39
  • 113
  • 193
7
votes
0 answers

kafka-python: Closing the kafka producer with 0 vs inf secs timeout

I am trying to produce the messages to a Kafka topic using kafka-python 2.0.1 using python 2.7 (can't use Python 3 due to some workplace-related limitations) I created a class as below in a separate and compiled the package and installed in virtual…
Anurag Rana
  • 1,429
  • 2
  • 24
  • 48
1 2 3
…
99 100