Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
0
votes
0 answers

Tracing context lost opentracing and brave for kafka

I've some flink jobs which uses kafka as source and sink and I want to add tracing to it, so that any message consumed/produced from/to Kafka is well traced, for that I'm using kafka interceptors to intercepts messages and log trace, span and parent…
Akhil
  • 1,184
  • 1
  • 18
  • 42
0
votes
1 answer

How can we convert a SourceRecord in Kafka Connect to AVRO?

I have a use-case where I need to convert a SourceRecord to a GenericRecord. Can someone help me how to do that? Basically, I'm writing a custom Kafka Producer and embed that in the Connect runtime code. When the producer.send() is called, the…
guru
  • 409
  • 4
  • 21
0
votes
1 answer

Kafka producer WARN

The following warning message observed in Kafka producer logs. I have a callback to notify failures. But didn't see failure logs. Is this WARN can cause a message failures to topic? 2020-02-17 10:23:47|[kafka-producer-network-thread |…
era
  • 391
  • 4
  • 24
0
votes
1 answer

Kafka Producer Integration Test Definition

I have an application code that produce events to a Kafka Topic. So now I needed to write Integration tests for it. After searching on internet, I got confused with definition of Integration test that whether should I use embedded Kafka server…
Vip
  • 1,448
  • 2
  • 17
  • 20
0
votes
1 answer

Kafka consumer receive null value when sending customer object

So i want to implement application which reads data from json format files. And I have created customer object for the data in json. And I want to send these object through kafka topic. So far i have successfully send String message to producer to…
ZHEN BIAN
  • 13
  • 1
  • 6
0
votes
1 answer

Kafka transaction - skip one offset when the application stops and be started again

To be fair, I have tested it with normal kafka without transaction scheme, and it does not skip the offset when I try to rerun the ProducerTest like a lot of times. object ProducerTest extends LazyLogging { def main(args: Array[String]): Unit = { …
alona
  • 35
  • 1
  • 1
  • 10
0
votes
0 answers

Python consumer script exits on server when no entry is made in kafka queue for more than a night

I have deployed Kafka and python consumers on separate servers. I am using GCP VM instances for deployment. Python scripts run in a loop and checks with kafka if it has a job for it. If it finds a job, it starts the operation otherwise it keeps…
Mayank
  • 1,364
  • 1
  • 15
  • 29
0
votes
1 answer

Kafka Connect with custom Kafka Producer

I'm trying to build a KafkaConnect source connector and was wondering if it is possible to inject/use a custom Kafka Producer in the process. As per the documentation, I couldn't get much. Any ideas would help!
guru
  • 409
  • 4
  • 21
0
votes
2 answers

Unable to run Kafka Console Producer (NoSuchMethodError)

Error while running kafka producer ./kafka-console-producer.sh --broker-list localhost:9092 --topic testing Exception in thread "main" java.lang.NoSuchMethodError:…
Abhishek Deepankar
  • 13
  • 4
0
votes
0 answers

Getting Kakfa producer to Talk through a tunnel

I have a use case in which I have a Kafka producer that needs to talk to a Kafka broker and a consumer, but with the added bonus of both the Broker and the Consumer being inside a customer's VPC (AWS VPC). How I have been looking at ways to get…
Arunav Sanyal
  • 1,708
  • 1
  • 16
  • 36
0
votes
1 answer

Kafka-Verifiable-Producer and Consumer Problem

I am doing experimenting with kafka. I already start the kafka-console-producer and kafka-console-consumer. I send messages with kafka-producer and successfully receive at the kafka-console-consumer. Now I want to produce and consume around 5000…
Rio
  • 347
  • 3
  • 6
  • 20
0
votes
1 answer

How to unit test the Kafka Streams and Producer APIs together

Currently, I have a basic Kafka streams application that involves a Topology with only a source and a processor, but no sink. Essentially, the Topology only handles the consumption of messages. As for producing messages, we make calls to the…
Chris Gong
  • 8,031
  • 4
  • 30
  • 51
0
votes
3 answers

Kafka Producer API: bootstrap.servers property value

I am learning Kafka Producer API, and inside tutorials, they have mentioned "bootstrap.servers" is mandatory property to specify currently running brokers(by comma-separated value). I doubted why the producer provides all the brokers' list why not…
ARPAN CHAKARVARTY
  • 377
  • 2
  • 10
0
votes
0 answers

KafkaTopicProvisioner failed to obtain partition

I observed my services going down with the below exception. The reason was one of our three Kafka brokers was down. And spring was always trying to connect with the same broker. Before it can skip faulty broker and connect to the next available…
Neeraj Kukreti
  • 141
  • 14
0
votes
2 answers

Where and how topic does get created in broker when script with details on number of partition and replica factor is given

when we create topic ,where we decide number of partition and replica factor. Do this topic get created in all the brokers? Is it specific to any one broker?
aayush mohanka
  • 33
  • 3
1 2 3
…
99 100