Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
0
votes
1 answer

Streaming database data to Kafka topic without using a connector

I have a use case where I have to push all my MySQL database data to a Kafka topic. Now, I know I can get this up and running using a Kafka connector, but I want to understand how it all works internally without using a connector. In my spring boot…
Nithin Prasad
  • 554
  • 1
  • 9
  • 22
0
votes
2 answers

How we can write single message(not batch) fast in kafka?

I am new to Golang and Kafka and I am using segmentio kafka-go to connect to Kafka server using Golang. As of now I want to push every event of user in Kafka, so I want to push single message(and not in batch), but since the write operation provided…
Suraj Singh
  • 3
  • 1
  • 3
0
votes
0 answers

How to send String data to kafka producer using Reactive Kafka?

I have a data in string format. I need to send it to kafka producer to a specific partition but using reactive kafka ie making async call. How to achieve this using code ? I checked reactive kafka docs but unfortunately I dont have much idea about…
aarish_codev
  • 43
  • 1
  • 5
0
votes
0 answers

Kafka & compression

i have 1 producer 1 consumer and 3 brokers in my set up. I am using python API for producer and consumer. all kafka parameters are set to default meaning no compression should occur at any kafka stage. when producer use "message = 'a' *…
ankit patel
  • 1,399
  • 5
  • 17
  • 29
0
votes
1 answer

How to measure number of send() request sent by Kafkaproducer in a batch of 2 mins

I am playing around the Kafka producer performance. And i need to reduce the number of requests that Kafka producer sends it to Kafka cluster by tuning batch size . I am using batch processing of 2 mins I tried printing metrics mentioned in…
LearnerForLife
  • 147
  • 1
  • 12
0
votes
0 answers

Error throwing while producing message to kafka topic using producer API?

I need to post the consumer record data to kafka topic using producer, While posting it was throwing an error : Magic v1 does not support add headers... I checked the broker version also and changed into old jar version. But it always shows an…
tisispa1
  • 203
  • 2
  • 3
  • 16
0
votes
0 answers

Timout Exception while sending data using Kafka Producer

I am getting this error, while producing data to UAT environmet: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for topic-2: 30808 ms has passed since batch creation plus linger time In the dev environment it is working fine,…
vbvT
  • 71
  • 1
  • 8
0
votes
1 answer

Kafka is trying to send messages to a broker in "recovery mode"

I have the following setup 3 Kafka (v2.1.1) Brokers 5 Zookeeper instances Kafka brokers have the following configuration: auto.create.topics.enable: 'false' default.replication.factor: 1 delete.topic.enable: 'false' …
Martin Linha
  • 979
  • 9
  • 21
0
votes
1 answer

Kafka connect Jdbc source connector data is stored as encoded string

I am new to kafka & exploring kafka connect in distributed mode. I have some issues which I have listed below. Data from my oracle table is stored as encoded values in strings. (for example, one of my column which is an integer has value 60015 is…
Bharath
  • 15
  • 7
0
votes
1 answer

How to dockerize and write a dockerfile a Scala written kafka producer

I am new to dockers and been trying to understand how to write a docker file to create my custom image. My Scala class is producing message to a topic continuously. I want to reproduce the same functionality with dockers. Can someone help me with…
Vaibhav Srivastava
  • 29
  • 1
  • 12
0
votes
3 answers

My producer can create a topic, but data doesn't seem to be stored inside the broker

My producer can create a topic, but it doesn't seem to store any data inside a broker. I can check that the topic is created with kafka-topics script. When I tried to consume with kafka-console-consumer, it doesn't consume anything. (I know…
Jin Lee
  • 3,194
  • 12
  • 46
  • 86
0
votes
1 answer

Does Java KafkaProducer open any sockets and pipes even when the KafkaProdcucer is not able to connect to kafka for some reason?

Does java KafkaProducer open any pipes or sockets to connect to kafka? What happens if kafka producer object is not able to connect to kafka, does it close these pipes and sockets?
rajukemburi
  • 31
  • 6
0
votes
0 answers

Connection leak from KafkaProducer

We are trying to achieve java KafkaProducer resiliency where in case Kafka is not reachable for some reason then we are storing failed messages in files and retrying once kafka comes up. But we are checking periodically to see if kafka is available…
rajukemburi
  • 31
  • 6
0
votes
1 answer

Kafka Producer Idempotence - Exactly Once or Just Producer Transaction is Enough?

From this article https://www.confluent.io/blog/transactions-apache-kafka/ Using vanilla Kafka producers and consumers configured for at-least-once delivery semantics, a stream processing application could lose exactly once processing semantics in…
user1955934
  • 3,185
  • 5
  • 42
  • 68
0
votes
1 answer

Not all attributes are available in kafka producer.metrics()

I am trying to get all the metrics of kafka producer via producer.metrics() api call. I am able to fetch few metrics but i am not able to see all the metrics mentioned in https://kafka.apache.org/20/documentation.html…
LearnerForLife
  • 147
  • 1
  • 12
1 2 3
…
99
100