Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
13
votes
1 answer

Is it possible to transfer files using Kafka?

I have thousands of files generated each day which I want to stream using Kafka. When I try to read the file, each line is taken as a separate message. I would like to know how can I make each file's content as a single message in Kafka topic and…
Nahush
  • 411
  • 1
  • 6
  • 16
13
votes
3 answers

Retry logic in kafka consumer

I have a use case where i consume certain logs from a queue and hit some third party API's with some info from that log , in case the third party system is not responding properly i wish to implement a retry logic for that particular log . I can add…
TECH007
  • 159
  • 1
  • 1
  • 6
12
votes
1 answer

Unexpected Kafka request of type METADATA during SASL handshake

I am trying to connect Kafka Java Client to a Kafka broker using SASL Plain. But when I try to send a message from the Producer, the Kafka Server logs the following error: [2020-04-30 14:48:14,955] INFO [SocketServer brokerId=0] Failed…
ThisaruG
  • 3,222
  • 7
  • 38
  • 60
12
votes
2 answers

No serviceName defined in either JAAS or Kafka config (not Kerberos)

I'm trying to configure a kafka client to authenticate against a secure kafkaserver. I've set up the jaas and ssl configs, but it's complaining about serviceNames. I am not using…
James Liu
  • 196
  • 1
  • 2
  • 10
12
votes
2 answers

Does kafka have any default web UI

I have a couple questions in Kafka. 1) Does Kafka have a default web UI? 2) How can we gracefully shutdown a standalone kafka server, kafka console- consumer/console-producer. Any solutions will be highly appreciated. Thank you.
Visweswaran Radhakrishnan
  • 149
  • 1
  • 1
  • 8
12
votes
1 answer

Meaning of sendOffsetsToTransaction in Kafka 0.11

The new Kafka version (0.11) supports exactly once semantics. https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging I've got a producer setup with kafka transactional code in java like this.…
Vishnu Prathish
  • 369
  • 4
  • 15
12
votes
2 answers

Kafka data types of messages

I was wondering about what types of data we could have in Kafka topics. As I know in application level this is a key-value pairs and this could be the data of type which is supported by the language. For example we send some messages to the topic,…
Fateax
  • 203
  • 1
  • 2
  • 11
12
votes
1 answer

Kafka optimal retention and deletion policy

I am fairly new to kafka so forgive me if this question is trivial. I have a very simple setup for purposes of timing tests as follows: Machine A -> writes to topic 1 (Broker) -> Machine B reads from topic 1 Machine B -> writes message just read to…
Mohammad Ahmad
  • 133
  • 1
  • 1
  • 8
12
votes
1 answer

Can I ignore org.apache.kafka.common.errors.NotLeaderForPartitionExceptions?

My Apache Kafka producer (0.9.0.1) intermittently throws a org.apache.kafka.common.errors.NotLeaderForPartitionException My code that performs the Kafka send resembles this final Future futureRecordMetadata = KAFKA_PRODUCER.send(new…
Hector
  • 4,016
  • 21
  • 112
  • 211
11
votes
9 answers

Unable to produce to Kafka topic that is running on WSL 2 from Windows

I am running the latest Kafka on Ubuntu WSL2 successfully. I can start zookeeper, kafka server, create topics, console produce and console consume just fine from within the Ubuntu that I have running on the WSL. However, when I go into my Intellij…
Viriato
  • 2,981
  • 7
  • 40
  • 54
11
votes
2 answers

How to list all producers of a kafka cluster?

I am able to list all Kafka Consumer with KafkaAdminClient: AdminClient client = AdminClient.create(conf); ListTopicsResult ltr = client.listTopics(); KafkaFuture> names = ltr.names(); ArrayList consumerGroups =…
kaiffeetasse
  • 481
  • 1
  • 8
  • 18
11
votes
1 answer

When does Kafka Leader Election happen?

When and how often does Kafka High Level Producer elect a leader? Does it do before sending each message or only once at the time of creating connection?
Marut Singh
  • 345
  • 1
  • 3
  • 10
11
votes
2 answers

Kafka Consumer's poll() method gets blocked

I'm new to Kafka 0.9 and testing some features I realized a strange behaviour in the Java implemented Consumer (KafkaConsumer). The Kafka broker is located in an Ambari external machine. Even thou I could implement a Producer and start sending…
aran
  • 10,978
  • 5
  • 39
  • 69
11
votes
1 answer

Kafka consumer "failed to find leader" when fetching topic metadata

When I try to use the Kafka producer and consumer (0.9.0) script to push/pull messages from a topic, I get the errors below. Producer Error [2016-01-13 02:49:40,078] ERROR Error when sending message to topic test with key: null, value: 11 bytes with…
Jordan Parmer
  • 36,042
  • 30
  • 97
  • 119
10
votes
1 answer

KafkaTimeoutError: Failed to update metadata after 60.0 secs

I have a use case of high throughput kafka producer where I want to push thousands of json messages every second. I have a 3 node kafka cluster and I am using latest kafka-python library and have following method to produce message def…
Sarvesh Kumar
  • 131
  • 1
  • 1
  • 4
1 2 3
…
99 100