Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
16
votes
7 answers

NoBrokersAvailable: NoBrokersAvailable-Kafka Error

i have already started to learn Kafka. Trying basic operations on it. I have stucked on a point which about the 'Brokers'. My kafka is running but when i want to create a partition. from kafka import TopicPartition (ERROR THERE) consumer =…
Beyhan Gul
  • 1,191
  • 1
  • 15
  • 25
15
votes
2 answers

Kafka - Broker: Message size too large

I get Message size too large exception, when I try to send a message which is over 1 Mb size. The error appears in my client application, when I try to produce a message. After a little googling I found out that the settings should be changed in…
managerger
  • 728
  • 1
  • 10
  • 31
15
votes
6 answers

How to choose the no of partitions for a kafka topic?

We have 3 zk nodes cluster and 7 brokers. Now we have to create a topic and have to create partitions for this topic. But I did not find any formula to decide that how much partitions should I create for this topic. Rate of producer is 5k…
Raj
  • 429
  • 2
  • 6
  • 26
15
votes
3 answers

Implications of keeping linger.ms at 0

We are using kafka 0.10.2.1. The documentation specifies that a buffer is available to send even if it isn't full- By default a buffer is available to send immediately even if there is additional unused space in the buffer. However if you want to…
Aditya Vivek
  • 553
  • 2
  • 5
  • 16
15
votes
2 answers

Apache Kafka Producer Config: 'request.timeout.ms' VS. 'max.block.ms' properties

Given the below synchronous kafka producer Properties props = new Properties(); props.put("max.block.ms", 30000); props.put("request.timeout.ms", 30000); props.put("retries", 5); KafkaProducer produce = new…
rugden
  • 1,085
  • 2
  • 9
  • 11
14
votes
3 answers

Is there a way to add headers in kafka-console-producer.sh

I'd like to use the kafka-console-producer.sh to fire a few JSON messages with Kafka headers. Is this possible? docker exec -it kafka_1 /opt/kafka_2.12-2.3.0/bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic…
tosi
  • 1,873
  • 2
  • 18
  • 38
14
votes
2 answers

How to determine API version of Kafka?

I am using kafka-python for accessing Kafka. I try to create a Kafka Producer: kafka_producer = KafkaProducer(bootstrap_servers=['kafka:9092']) but this fails with exception kafka.errors.NoBrokersAvailable: NoBrokersAvailable. I've found out I need…
Michal Špondr
  • 1,337
  • 2
  • 21
  • 44
14
votes
3 answers

Kafka: The message when serialized is larger than the maximum request size you have configured with the max.request.size configuration

Getting the following error (Kafka 2.1.0): 2018-12-03 21:22:37.873 ERROR 37645 --- [nio-8080-exec-1] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='{82, 73, 70, 70, 36, 96, 19,…
sapy
  • 8,952
  • 7
  • 49
  • 60
14
votes
4 answers

On kafka console not able to type message with size more than 4095 characters

I am trying to send message through kafka console producer. But I am not able to type messages more than 4095 characters. Tried to search if there is any property related to this in producer or server but to not avail. Even tried to search if…
user2714361
  • 141
  • 1
  • 4
14
votes
4 answers

Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?

I have setup a kafka zookeeper and 3 brokers on one machine on ec2 with ports 9092..9094 and am trying to consume the topic content from another machine. The ports 2181 (zk), 9092, 9093 and 9094 (servers) are open to the consumer machine. I can even…
notrai
  • 696
  • 1
  • 5
  • 15
13
votes
2 answers

Kafka producer is connecting to localhost instead of the real IP

Please help I'm wondering why the kafka producer always connect to the localhost however there the broker ip is not the localhost. So, is there any help ? any ideas ? import org.apache.kafka.clients.producer.KafkaProducer; import…
Farag Zakaria
  • 178
  • 1
  • 9
13
votes
4 answers

What is reason for getting ProducerFencedException during producer.send?

Trying to load around 50K messages into KAFKA topic. In the beginning of few runs getting below exception but not all the time. org.apache.kafka.common.KafkaException: Cannot execute transactional method because we are in an error state at…
Krishna
  • 393
  • 2
  • 3
  • 18
13
votes
1 answer

Is there a way to set a delay for a message sent by a kafka producer?

Or maybe even a way to delay the message received by the consumer. I need to make a function call in nodejs after every 90s, so I want to add a delay of 90s for every kafka message
Samarth Juneja
  • 766
  • 1
  • 8
  • 21
13
votes
2 answers

Kafka Producer: Got error produce response with correlation NETWORK_EXCEPTION

We are running kafka in distributed mode across 2 servers. I'm sending messages to Kafka through Java sdk to a Queue which has Replication factor 2 and 1 partition. We are running in async mode. I don't find anything abnormal in Kafka logs. Can…
Anil Kumar
  • 2,521
  • 7
  • 23
  • 40
13
votes
6 answers

Start Confluent Schema Registry in windows

I have windows environment and my own set of kafka and zookeeper running. To use custom objects, I started to use Avro. But I needed to get the registry started. Downloaded Confluent platform and ran this: $ ./bin/schema-registry-start…
user1860447
  • 1,316
  • 8
  • 25
  • 46
1 2
3
…
99 100