Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
0
votes
0 answers

Two Kafa consumer with one producer

I have two kafka consumers defined in two micro services from one producer with different group id defined.One of the micro service is not consuming the message from the producer @Component @ConditionalOnProperty(value = "XXX", havingValue =…
ASP
  • 1
  • 2
0
votes
1 answer

Less partitions than consumers in kafka, how to Optimize other consumers of that consumer group

Let's assume the scenario where I have four partitions and a consumer group having twenty consumers. So only four consumers will be working and sixteen will be in starvation mode. How to optimize that sixteen other consumers so we will have every…
ANUJ PATEL
  • 1
0
votes
1 answer

How to run kafka producer.sh & consumer.sh for a particular user to test ACL?

./bin/kafka-console-producer.sh --broker-list 10.162.0.6:32294 --topic my-topic ./bin/kafka-console-consumer.sh --bootstrap-server 10.162.0.6:32294 --topic my-topic --from-beginning Am new to kafka. My kafka running on strimzi kubernetes operator…
sudhir tatraju
  • 93
  • 1
  • 7
0
votes
1 answer

How to add data to a topic using AvroProducer

I have a topic with the following schema. Could someone help me out on how to add data to the different fields. { "name": "Project", "type": "record", "namespace": "abcdefg", "fields": [ { "name": "Object", "type": { …
Aerluft
  • 41
  • 1
  • 7
0
votes
1 answer

Does Kafka Preferred leader gets updated ever, even if the brokers are all up?

I am trying to make a kafka monitoring service, which needs to have at least one partition of a topic on each broker in the cluster. I assigned the partitions initially. I have set the replication factor to 1, and min.insync.replicas is also 1, and…
srnand
  • 1
0
votes
2 answers

How can i try to read to topic in kafka when i am getting an error as Missing required argument "[topic]"

I am getting an error while reading a topic from a single node cluster kafka-console-producer.bat -- broker-list localhost:9092 -- topic raja Error below Missing required argument "[topic]" kafka version-kafka_2.13-2.4.0
Rockyy
  • 7
  • 1
  • 6
0
votes
1 answer

Kafka consumer's first poll does not retrieve the topic messages. What could be wrong?

I have a simple Kafka 2.4.1 (Confluent 5.4.1) setup running locally in Docker. And I use a test producer and a test consumer written in Java. The code is available in GitHub. The unit tests do: a producer produces one message to a…
Gabriel Petrovay
  • 20,476
  • 22
  • 97
  • 168
0
votes
0 answers

Calculating Kafka Partitions to achieve a specifc throughput

If producer throughput is 5 mb/sec and consumers is 15mb/sec , I want to achieve an overall throughput of 75 mb/sec. Is the right calculation rule for the number of partitions MAX(throughput/prod-throughput , throughput/consumer-throughput) ? or…
Muhammad Magdi Youssif
  • 107
  • 6
0
votes
1 answer

cant read\write from kafka topic

I have Kafka running on a container on my desktop. I can connect to it just fine using a tool called "Kafka tool" where I can see my topics for example. I'm having issues reading and writing to/from a Kafka topic. what's annoying me is that it…
Diego
  • 34,802
  • 21
  • 91
  • 134
0
votes
1 answer

Not able to Produce Message to Kafka topic using Transactional.Sink in Alpakka but I see idempotent producer is enabled

Hi I was trying to use Producer api like shown in Alpakka documentation. I'm able to consume record using Transactional source and Producer is created but not able to put message to topic Not able to Produce to topic using Transactional.Sink in…
Appy
  • 63
  • 1
  • 4
0
votes
1 answer

Do Kafka's parallelism concepts apply when running in console mode?

So, I am new to Kafka and I have been reading about it for a while. I found this information on confluent. https://docs.confluent.io/current/streams/architecture.html So what I understood from this is, say I have a topic called plain_text where I…
Shubhang b
  • 597
  • 1
  • 5
  • 12
0
votes
0 answers

Cannot connect to kafka endpoint from local machine from spark streaming

Here is my spark streaming code. val spark = SparkSession .builder .appName("StructuredWatermarkingWC") .master("local[*]") .config("spark.sql.streaming.checkpointing","/tmp/") .getOrCreate() …
Srinivas
  • 2,010
  • 7
  • 26
  • 51
0
votes
1 answer

What defines the scope of a kafka topic

I'm looking to try out using Kafka for an existing system, to replace an older message protocol. Currently we have a number of types of messages (hundreds) used to communicate among ~40 applications. Some are asynchronous at high rates and some…
cojae
  • 27
  • 4
0
votes
1 answer

Spring Kafka Consumer ACKMODE & Producer buffering for Kafka transactions

I need to use the consume-process-produce pattern for processing Kafka messages and have configured a Spring Kafka listener container with a Kafka Transaction Manager and have also set the transaction-id-prefix to enable Kafka transactions. I am…
ankush
  • 167
  • 1
  • 11
0
votes
3 answers

How to save in local storage messages when Kafka Producer does not send a message?

I have 120 data producers with Kakfa Producer, they send messages each second and some have sensible data (i mean, we have to send it yes or yes), also these producers sometimes could shutdown or lose internet connection, so I need fault tolerance…
O. Azofeifa
  • 1
  • 1
1 2 3
…
99 100