Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
7
votes
1 answer

Kafka Producer Error: ' Value serializer not specified and there is no default serializer defined for type ...'

I just started using Kafka and hit the following rookie error: 'Value cannot be null. Parameter name: Value serializer not specified and there is no default serializer defined for type ActMessage.' It happens when trying to send a class object,…
MiguelSlv
  • 14,067
  • 15
  • 102
  • 169
7
votes
1 answer

Python librdkafka producer perform against the native Apache Kafka Producer

I am testing Apache Kafka Producer with native java implementation against Python's confluent-kafka to see which has the maximum throughput. I am deploying a Kafka cluster with 3 Kafka brokers and 3 zookeeper instances using docker-compose. My…
Amit Tripathi
  • 7,003
  • 6
  • 32
  • 58
7
votes
3 answers

Kafka throttle producer based on consumer lag

Is there any way to pause or throttle a Kafka producer based on consumer lag or other consumer issues? Would the producer need to determine itself if there is consumer lag then perform throttling itself?
mrmannione
  • 749
  • 10
  • 29
7
votes
2 answers

Producing a Kafka message with a Null Value (Tombstone) from the Console

Is there any way to produce a message in the kafka-console-producer with a null value (ie. mark it for the compactor to delete it with a tombstone)? I've tried producing "mykey" and "mykey|". The former produces an error and the later makes the…
David Seapy
  • 73
  • 1
  • 1
  • 4
7
votes
7 answers

Uneven Distribution of messages in Kafka Partitions

I have a topic with 10 partitions, 1 consumer group with 4 consumers and worker size is 3. I could see there is an uneven distribution of messages in the partitions, One partition is having so much data and another one is free. How can I make my…
Pacifist
  • 3,025
  • 2
  • 12
  • 20
7
votes
2 answers

how to get last committed offset from read_committed Kafka Consumer

I am using the transactional KafkaProducer to send messages to a topic. This works fine. I use a KafkaConsumer with read_committed isolation level and I have an issue with the seek and seekToEnd methods. According to the documentation, the seek and…
Coen Damen
  • 2,009
  • 5
  • 29
  • 51
7
votes
3 answers

How do I install an extension of Kafka for PHP?

This is the extension that I am trying to install: https://github.com/EVODelavega/phpkafka The messages passed to the queue should be in JSON format. Currently, I am getting installation errors: 1. The instructions ask me to install librdkafka. 2.…
Aviral Srivastava
  • 137
  • 1
  • 2
  • 6
7
votes
3 answers

Apache Kafka and Avro: org.apache.avro.generic.GenericData$Record cannot be cast to com.harmeetsingh13.java.Customer

Whenever I am trying to read the message from kafka queue, I am getting following exception : [error] (run-main-0) java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to…
Harmeet Singh Taara
  • 6,483
  • 20
  • 73
  • 126
7
votes
2 answers

Is KafkaTemplate thread safe

Is KafkaTemplate in spring boot thread safe. Can I create one KafkaTemplate and use it for sending information to the same kafka topic for multiple requests in my web service.
IS_EV
  • 988
  • 2
  • 15
  • 29
7
votes
2 answers

Desigining Kafka Topics - Many Topics vs One Big Topic

Considering a stream of different events the recommended way would be one big topic containing all events multiple topics for different types of events Which option would be better? I understand that messages not being in the same partition of a…
user3452075
  • 411
  • 1
  • 6
  • 17
7
votes
2 answers

spark kafka producer serializable

I come up with the exception: ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Task not serializable org.apache.spark.SparkException: Task not serializable at …
Steven.Prgm
  • 73
  • 1
  • 5
7
votes
1 answer

KafkaProducer not successfully sending message into the queue

I have built a small testing environment on my Windows PC and write down the following code for testing kafka (using kafka_2.10:0.9.0.1 from org.apache.kafka). package iii.functiontesting; import java.text.ParseException; import…
greencomet
  • 103
  • 1
  • 2
  • 9
7
votes
2 answers

How is ordering guaranteed during failures in Kafka Async Producer?

If I am using Kafka Async producer, assume there are X number of messages in buffer. When they are actually processed on the client, and if broker or a specific partition is down for sometime, kafka client would retry and if a message is failed,…
Guruprasad GV
  • 916
  • 13
  • 18
7
votes
1 answer

Java consumer group missing?

I recently set up a test Kafka cluster. I am running a consumer group listening on items and things seem to work. The name of consumer group is default. What surprises me is that listing consumer groups gives me an empty list: $…
Ztyx
  • 14,100
  • 15
  • 78
  • 114
6
votes
1 answer

Does enabling Idempotence on a Kafka producer decrease throughput

I have kafka producer with idempotence enabled (without exactly once semantics or transaction enabled) within a rest endpoint call. The reason I enabled it is because the I did not want any duplicates cause by the kafka retries. I am concerned about…
Smalltalkguy
  • 319
  • 1
  • 4
  • 13
1 2 3
…
99 100