Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
0
votes
1 answer

KAFKA : splitting and retrying (1 attempts left). Error: MESSAGE_TOO_LARGE

I am sending 10 messages . 2 messagesare "right" and 1 message has size over 1MB which gets rejected by Kafka broker due to RecordTooLargeException. I have 2 doubts 1) MESSAGE_TOO_LARGE appears only when the Scheduler calls the method second time…
Pale Blue Dot
  • 511
  • 2
  • 13
  • 33
0
votes
1 answer

Wait for List of ListenAbleFuture returned by Kafka Send API

I have the List of ListenAbleFuture.I want to wait of this List of ListenableFuture> for atmost 15 minutes if they have not comepleted. How can i achieve it. Currently i am doing this but this wait for 15 min for every…
Pale Blue Dot
  • 511
  • 2
  • 13
  • 33
0
votes
1 answer

Wait for kafkaTemplate pending future

I am performing an asynchronous operation which returns a future object in a loop (say 10 messages) . As I understand callback method is automatically triggered and executed when the Future completes its task. Suppose my 7th future is in pending…
Pale Blue Dot
  • 511
  • 2
  • 13
  • 33
0
votes
1 answer

Exception while connecting kafka server through Pepperbox sampler in jmeter

I am using pepprBox Sample in JMeter to connect kafka server and sending messages on topic. Application team shared two JKS certificate along with password to connect Kafka server. For this, I have enabled keystore property in system.properties…
jayesh
  • 3
  • 1
0
votes
1 answer

Getting intermittent KafkaProducerException: Failed to send org.apache.kafka.common.errors.TimeoutException

I am getting below error - Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) as below, Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for pipeline-demo-0: 60125 ms has passed…
Alaukik Srivastava
  • 41
  • 1
  • 5
0
votes
1 answer

How to handle Kafka schema evolution

I'm new to kafka .Here is the question i have on ever changing kafka schema. How can we handle schema changes at kafka consumer end? If we change the payload structure at kafka publisher end, how can I make sure nothing breaks on the kafka consumer…
MonkTechie
  • 33
  • 5
0
votes
1 answer

Kafka file stream connect and stream API

am working on the file stream connector, I have more than ten million records in the file(it's not a single file, its partition by account #). I have to load these files into the topic and update my streams. have gone through stand-alone streams, I…
Naresh Govindaswamy
  • 41
  • 3
0
votes
1 answer

Kafka Node crashes intermittently when data is pushed from producer

We have a 3 node Kafka cluster (version 5.2.1, apache kafka version: 2.2.0) in our environment. For sometime we have been observing an exception which happens intermittently whenever we try to push data from a test producer. Following is the…
user1184527
  • 113
  • 1
  • 12
0
votes
1 answer

kafka asynchronous produce lost message

Try to follow the instruction on internet to achieve kafka asynchronous produce. Here is what my producer looks like: import org.apache.kafka.clients.producer.Producer; import org.apache.kafka.clients.producer.ProducerRecord; public void…
Fallin
  • 45
  • 10
0
votes
2 answers

Kafka topics exist but not available from the brokers

I have over 1,000 topics. After an outage I find that only a handful of topics are showing up when listing the topics. If I write to an existing but unavailable topic it says: [2020-04-17 16:17:09,153] WARN [Producer clientId=console-producer]…
Earl Bunch
  • 1
  • 1
0
votes
1 answer

dynamic switching between two kafkaTemplate objects

I have two kafka clusters (Active-passive). I created two KafkaTemplate objects for producing events(one for each). I will keep using primary kafkaTemplate to send events as long as it is healthy. Incase of any issue, i want to dynamically switch…
user13338722
0
votes
3 answers

How to pass user created to producer.sh/consumer.sh in Kubernetes Strimzi Kafka Operator cluster?

I deployed Strimzi operator for Kafka & enabled TLS/ACLs, now I am stuck! How to run producer.sh/consumer.sh on 9093 passing user created? Below is the command working with ACLs disabled on port 9092: kubectl -n myproject run kafka-producer -ti…
sudhir tataraju
  • 1,159
  • 1
  • 14
  • 30
0
votes
1 answer

Producer metric: record_error_rate ,when will this value be greater than zero

I am checking and validating the metrics of producer and consumers. But metric record_error_rate is always seems to be zero in all my experiments. If someone has any ideas when will this be non-zero then please let me know, also the steps to…
dvsakgec
  • 3,514
  • 4
  • 28
  • 35
0
votes
1 answer

Multiple instances of kafka producer

I understand that multiple instances of a producer application can be started to produce data to a topic. What configuration changes are required for this? Do i need different client_id? Without any changes , it works. So not sure if there are any…
Sanjay
  • 165
  • 1
  • 13
0
votes
1 answer

Confluent kafka Python client Avro producer.producer() executes without error but no data in topic

My producer isnt throwing any errors but data is not being sent to the destination topic. Can you recommend any techniques to debug this situation. I have call to a Confluent Python Avro Producer inside a synchronous loop to send data to a topic…
banditKing
  • 9,405
  • 28
  • 100
  • 157
1 2 3
…
99 100