Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
6
votes
3 answers

Excessive console messages from Kafka Producer

How do you control the console logging level of a Kafka Producer or Consumer? I am using the Kafka 0.9 API in Scala. Every time send on the KafkaProducer is called, the console gives output like below. Could this indicate I do not have the…
Peter Becich
  • 989
  • 3
  • 14
  • 30
6
votes
1 answer

Converting pojos to generic records in confluent.io to send through a KafkaProducer

I am completely new to Kafka and avro and trying to use the confluent package. We have existing POJOs we use for JPA and I'd like to be able to simply produce an instance of my POJOs without having to reflect each value into a generic record…
Joe
  • 71
  • 5
6
votes
0 answers

Seeing "partition doesn't exist" warnings/failures after kafka using kafka partition re-assignment tool

I am using kafka 0.8.1.1. I have a 3 node kafka cluster with some topics having around 5 partitions. I planned to increase the number of nodes to 5 in cluster and moving some partitions from existing topics to the new brokers. Previous partition…
surekav
  • 83
  • 1
  • 4
5
votes
1 answer

Kafka producer property enable.idempotence=true is causing error

On setting kafka producer property - enable.idempotence to true kafkaProps.put("enable.idempotence" , "true"); I am getting below error - 2021-04-18 16:43:53.584[0;39m [31mERROR[0;39m [35m15524[0;39m [2m---[0;39m [2m[ad | producer-1][0;39m…
joven
  • 371
  • 1
  • 6
  • 17
5
votes
2 answers

Kafka message size with activated compression

i'm a little confused about the message size configuration in Kafka 2.6.0. But let's tell the story: We are using a Kafka cluster consisting of 3 nodes. So far with the standard configuration for messages. "zstd compression" is activated. The…
Christian Kittel
  • 81
  • 1
  • 6
5
votes
2 answers

What is the difference between Kafka Template and kafka producer?

As I have seen the Kafka template internally used Kafka producer. I just want to know what is the exact difference. Also, I found many send() methods available in the Kafka template as compared to Kafka producer. Please help me with it. If anyone…
Aadi
  • 1,131
  • 2
  • 16
  • 32
5
votes
1 answer

Java Producer/Consumer kafka client properties required when accessing a SSL-Auth secured Kafka brokers/cluster?

When we connect to Kafka cluster/kafka, In the java clients we define certain properties - Example Producer Properties Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("acks", "all"); …
samshers
  • 1
  • 6
  • 37
  • 84
5
votes
0 answers

Kafka Connection to 2 was disconnected before the response was read

We are getting the below exception in Kafka server.Please let me know why we are getting this issue? This issue keep on coming in the production environment. kafka version is 2.12-2.3.0 rEpoch=Optional[2708]), rds_exception-1=(fetchOffset=0,…
user3094331
  • 532
  • 6
  • 21
5
votes
1 answer

How to produce Kafka messages with JSON format in Python

How to delete quotes and send data like on original format The original JSON-format is: { "@timestamp": "2020-06-02T09:38:03.183186Z" } This data in another topic "{\"@timestamp\": \"2020-05-25T17:40:47.582778Z\"}" This is a code of sending data…
LOTR
  • 113
  • 1
  • 1
  • 10
5
votes
0 answers

How to produce Kafka Events in Django the right way

I am using django for a project and want to trigger a Kafka event on some request. I know how to produce a kafka event in a view. class TestList(generics.ListCreateAPIView): permission_classes = [permissions.IsAuthenticated] queryset =…
Greeneco
  • 691
  • 2
  • 8
  • 23
5
votes
1 answer

How to write Kafka Producer in Scala

I need help in publishing a message to a topic using kafka producer. My kafka producer client is written in scala running over spark. My Job runs successfully but it seems my message is not published. Here is the code val response =…
Deepak Garg
  • 149
  • 1
  • 10
5
votes
3 answers

Kafka Log Compacted Topic Duplication Values against same key not deleted

Log compacted topics are not supposed to keep duplicates against the same key. But in our case, when a new value with the same key is sent, the previous one isn't deleted. What could be the issue? val TestCompactState: KTable[String, TestCompact] =…
Danish
  • 257
  • 2
  • 12
5
votes
1 answer

Kafka Producer From Remote Server

I'm developing a streaming API with Apache Kafka version (2.1.0). I have a Kafka cluster and an external server. The external Server will be producing data to be consumed on the Kafka cluster. Let's denote the external Server as E and the cluster…
Joe jim
  • 59
  • 3
5
votes
3 answers

How to dynamically add consumers in consumer group kafka

How should I know when i have to scale the consumer in consumer group . What are the triggers for the consumers to scale when there is a fast producer ?
Vamsi Jakkula
  • 71
  • 1
  • 5
5
votes
1 answer

Cannot Connect from Kafkacat running in docker to Kafka broker running locally on windows machine

I am running kafka locally on a windows machine. The way I am running kafka is using .\bin\windows\kafka-server-start.bat .\config\server.properties The server.properties…
Zein Sleiman
  • 254
  • 2
  • 11
1 2 3
…
99 100