Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
0
votes
2 answers

Is there a way to find out the time taken for serialisation of data by Kafka

I want to time the time taken by Kafka to serialize different data formats. And have a doubt whether I can do it on my end(since I think this is done on the Kafka side.) If yes how can we do it?Is the serialisation done after the…
humble_me
  • 331
  • 3
  • 12
0
votes
1 answer

Messages lost while setting up Kafka mirror for disaster recovery server

Producer.config key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer max.in.flight.requests.per.connection=1 retries=3 acks=-1 Steps: started Kafka mirror on…
Amita Rawat
  • 153
  • 1
  • 2
  • 6
0
votes
2 answers

Distribute messages on single Kafka topic to specific consumer

Avro encoded messages on a single Kafka topic, single partitioned. Each of these messages were to be consumed by a specific consumer only. For ex, message a1, a2, b1 and c1 on this topic, there are 3 consumers named A, B and C. Each consumer…
Lawrence Leung
  • 305
  • 1
  • 11
0
votes
2 answers

how to send message remotely to kafka

I am very new to kafka. I am trying to send a message from my local machine producer to kafka server. I am not able to figure out the issue or what am doing worng. $config = \Kafka\ProducerConfig::getInstance(); …
int_ashish
  • 121
  • 1
  • 5
0
votes
1 answer

Unable to solve the error: java.io.NotSerializableException: org.apache.avro.Schema$RecordSchema

I am trying to read data from a table through SparkSession, and publish it to a Kafka topic. Using below piece of code for the same: import org.apache.avro.Schema import org.apache.avro.generic.{GenericData, GenericDatumWriter,…
Mohit Sudhera
  • 341
  • 1
  • 4
  • 16
0
votes
1 answer

Does Kafka provides Producer level offset?

Suppose , I've got multiple Kafka producers producing data simultaneously to a single Kafka Topic. Is it possible to get which was the last offset produced by a given producer ? For example : Producers : P1 , P2 Topic : T1 with one partition…
charany1
  • 871
  • 2
  • 12
  • 27
0
votes
1 answer

How to poll kafka for the messages that has timestamp less than 30 seconds?

I want to poll only the records that have produced less than 30 seconds. Is there any way in the Kafka polling that I can consume only the messages that are produced only less than 30 seconds? Any suggestions or advice will be really helpful. Thanks…
Manoj
  • 1,833
  • 3
  • 14
  • 11
0
votes
1 answer

package kafka.producer does not exist after changing to kafka_2.11

I have changed to Kafka version from 2.10 to 2.11 in pom file, now I am getting below error changes in pom from org.apache.kafka .…
user2232840
  • 63
  • 1
  • 7
0
votes
1 answer

how to use globalKtable and StateStore on the same topic?

Just to clarify, I'm new to Kafka, so sorry if my questions seems undocumented, I am reading tutorials, docs and everything I can to understand this. I am trying to read all values from a GlobalStore in order to update it's values, then use the…
Russo
  • 301
  • 3
  • 12
0
votes
1 answer

Spring @Transactional issue between Kafka and DB call

I am trying to check the Transactional state between Kafka and DB call(Postgres) where if the DB call gets failed then Kafka should not send that particular message onto the Kafka topic. I was trying to achieve this with the help of @Transactional…
Rahul Vel
  • 35
  • 1
  • 7
0
votes
0 answers

How to avoid transaction expiry in Kafka Streams application with idle threads

I am using Kafka Streams in a critical application, and I am facing issues where transactions are getting expired in idle threads. This causes issues after re-balance where a task shifts to a previously idle thread where the producer has expired.…
user10418702
  • 11
  • 1
0
votes
1 answer

How to secure kafka Topic with username and password from CLI/command line?

I have installed Docker on my Windows 10 and also installed Kafka. I have created a "test" Topic inside a Kafka cluster. Now I want to secure the Topic with a simple username and password. I am super new to Kafka, any help would really be…
kishorK
  • 453
  • 2
  • 7
  • 16
0
votes
1 answer

Getting TimeoutException for some messages while sending to Kafka topic

Exception Stacktrace: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ****-656 due to 30037 ms has passed since batch creation plus…
amitwdh
  • 661
  • 2
  • 9
  • 19
0
votes
1 answer

Add header while connecting to schema registry

I want to send some custom request headers while connecting to schema registry from my java producer application. Everytime I connect to the registry using the app, I want to send a header with the app-name so as to authenticate if the app can use…
mythic
  • 535
  • 7
  • 21
0
votes
1 answer

How to efficiently produce messages out of a collection to Kafka

In my Scala (2.11) stream application I am consuming data from one queue in IBM MQ and writing it to a Kafka topic that has one partition. After consuming the data from the MQ the message payload gets splitted into 3000 smaller messages that are…
Michael Heil
  • 16,250
  • 3
  • 42
  • 77
1 2 3
…
99 100