Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
0
votes
0 answers

Kafka cluster unavailability detection (3 brokers)

I have a Apache Kafka cluster with 3 brokers and I would like to detect when the cluster is no longer available in order to switch the client connection to a second replicated cluster (as described here: How to consume from two different clusters in…
Stefan
  • 21
  • 1
  • 4
0
votes
1 answer

Python, Kafka and Docker - KafkaConsumer keeps hanging

I have the following docker-compose file: version: '3.1' services: postgres_db: image: postgres restart: always environment: POSTGRES_USER: admin POSTGRES_PASSWORD: admin POSTGRES_DB: default_db …
Omri Shneor
  • 965
  • 3
  • 18
  • 34
0
votes
1 answer

Spring for Apache Kafka: Unrecognized producer configuration - 'delivery.timeout.ms'

I have a spring boot application with Spring for Apache Kafka being used to send messages to topics on a kafka cluster. I've autowired a property for delivery.timeout.ms and after the application starts and the producer is configured, the…
HendPro12
  • 1,094
  • 3
  • 17
  • 50
0
votes
1 answer

Standalone Kafka Producer

I'm thinking about creating a stand alone Kafka producer that runs as a daemon and takes messages via a socket and send them reliable to Kafka. But, I must not be the first one to think about this idea. The idea is to avoid writing a Kafka producer…
Sergeant
  • 131
  • 1
  • 11
0
votes
0 answers

How to send JAVA objects to kafka using compression provided by KAFKA?

So I am trying to send an object of around 5 MB to Kafka and I intend to use Kafka provided compression for it. I have not changed any default properties. My producer properties are:- public static Producer createProducer() { …
Tushar Yadav
  • 55
  • 1
  • 6
0
votes
1 answer

Kafka serialization difference between LongSerializer and Serdes.LongSerde

We are writing a new application which publish to Kafka and we need to serialize the key of the messages as a long value. When checking the Kafka docs for serialization, seems there are two long serializer classes as LongSerializer and…
Bathiya Priyadarshana
  • 1,325
  • 6
  • 22
  • 35
0
votes
0 answers

Apache Kafka for .Net Clients Non-Confluent Packages

I currently want to be able to have my Kafka Clients written in .Net. Although Java/Scala tend to be the most popular methods of writing Kafka APIs, my company uses Cloudera for support for Apache Kafka which does not offer .Net Support. Confluent…
Kevin Jang
  • 1
  • 1
0
votes
2 answers

Is it possible to publish xml in kafka?

Is it possible to publish xml in kafka? If yes, which method can be used to serialize data to be published in kafka?
anuj trehan
  • 95
  • 1
  • 4
  • 10
0
votes
0 answers

Kafka stream throws RecordTooLargeException exception on forwarding message from Processor

Since we migrated to Kafka-client 2.3 (previously it was 1.1) we are getting RecordTooLargeException exception intermittently when message is sent to Kafka topic. max.request.size value for the producer is 524288. As you can see below Event object…
Dima Svider
  • 377
  • 2
  • 15
0
votes
0 answers

Kafka authorization and authentication ports

I have read the documentation on Kafka security here: https://kafka.apache.org/documentation/#security_authz_cli and I was wondering something about the ports they use. For the authentication portion, under 7.2 Encryption and authentication using…
tom dinh
  • 143
  • 1
  • 4
  • 13
0
votes
1 answer

What is the usage of id property of @KafkaClient

I am new in Kafka and microservices. I tried to understand the usage of id property of @KafkaClient but most of the examples are given without that property therefore could not understand clearly. @KafkaClient(id = "client_id") What is the usage…
user1474111
  • 1,356
  • 3
  • 23
  • 47
0
votes
1 answer

Kafka topic not received some records from publisher send()

A Java application sends or publishes messages to Kafka cluster. Code to send records is given below. As this application is sending huge volumes of record messages to kafka, it is a bit difficult to debug. But I see some of the messages are not…
Nomad
  • 751
  • 4
  • 13
  • 34
0
votes
1 answer

Caused by: java.io.NotSerializableException: org.apache.kafka.clients.producer.KafkaProducer

I try to send the java String messages with kafka producer. And String messages are extracted from Java spark JavaPairDStream. JavaPairDStream processedJavaPairStream = inputStream.mapToPair (record-> new…
Joseph Hwang
  • 1,337
  • 3
  • 38
  • 67
0
votes
1 answer

How to create chunks of large message (above 1MB) in Kafka stream producer using nodejs or java?

I tried to create message chunks of Kafka producer in nodejs using kafka-node but didn't get any solution. so now I am creating a Kafka stream producer using Java and I need to send large message which has size above 1MB. How can I create chunks of…
Vaibhav Shelar
  • 75
  • 1
  • 2
  • 8
0
votes
0 answers

Avro messages not showing up in All Partitions

Avro messages are showing up in specific partition but not showing up in all Partitions.
Ashish Garg
  • 19
  • 2
1 2 3
…
99
100