Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
51
votes
1 answer

What is the difference in Kafka between a Consumer Group Coordinator and a Consumer Group Leader?

I see references to Kafka Consumer Group Coordinators and Consumer Group Leaders... What is the difference? What is the benefit from separating group management into two different sets of responsibilities?
Jeff Widman
  • 22,014
  • 12
  • 72
  • 88
50
votes
2 answers

Kafka Consumer get key value pair

I'm currently working with Kafka and Flink, I have kafka running in my local PC and I created a topic that is being consumed. Desktop\kafka\bin\windows>kafka-console-consumer.bat --zookeeper localhost:2181 -topic test but it is just retrieving the…
André De La O Campos
  • 1,027
  • 1
  • 8
  • 19
50
votes
4 answers

How to write a file to Kafka Producer

I am trying to load a simple text file instead of standard input in Kafka. After downloading Kafka, I performed the following steps: Started zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties Started…
Katie
  • 763
  • 1
  • 9
  • 21
46
votes
6 answers

understanding consumer group id

I did fresh installation of Apache Kafka 0.10.1.0. I was able to send / receive messages on command prompt. While using Producer / Consumer Java Example, I am not able to know group.id parameter on Consumer Example. Let me know on how to fix this…
Ankit
  • 2,126
  • 4
  • 33
  • 53
44
votes
6 answers

How can I get the last/end offset of a kafka topic partition?

I'm writing a kafka consumer using Java. I want to keep the real time of the message, so if there are too many messages waiting for consuming, such as 1000 or more, I should abandon the unconsumed messages and start consuming from the last…
Rinze
  • 706
  • 1
  • 5
  • 21
44
votes
15 answers

Consumer not receiving messages, kafka console, new consumer api, Kafka 0.9

I am doing the Kafka Quickstart for Kafka 0.9.0.0. I have zookeeper listening at localhost:2181 because I ran bin/zookeeper-server-start.sh config/zookeeper.properties I have a single broker listening at localhost:9092 because I…
EthanP
  • 1,663
  • 3
  • 22
  • 27
42
votes
2 answers

Kafka consumer for multiple topic

I have a list of topics (for now it's 10) whose size can increase in future. I know we can spawn multiple threads (per topic) to consume from each topic, but in my case if the number of topics increases, then the number of threads consuming from the…
Akhil
  • 1,184
  • 1
  • 18
  • 42
42
votes
2 answers

How does consumer rebalancing work in Kafka?

When a new consumer/brorker is added or goes down, Kafka triggers a rebalance operation. Is Kafka Rebalancing a blocking operation. Are Kafka consumers blocked while a rebalancing operation is in progress?
java_geek
  • 17,585
  • 30
  • 91
  • 113
40
votes
8 answers

Kafka Consumer default Group Id

I'm working with Apache Kafka and its Java client and I see that messages are load balanced across different Kafka Consumers belonging to the same group (i.e. sharing the same group id). In my application I need all consumers to read all…
Andrea Rossi
  • 981
  • 1
  • 10
  • 23
40
votes
3 answers

Kafka - difference between Log end offset(LEO) vs High Watermark(HW)

What is the difference between LEO and HW in Replica ( Leader Replica)? Will they contain the same number? I can understand HW is the last committed message offset. When LEO will be updated and how?
Shankar
  • 8,529
  • 26
  • 90
  • 159
40
votes
5 answers

How to create topics in apache kafka?

What is the bestway to create topics in kafka? How many replicas/partitions to be defined when we create topics? In the new producer API, when i try to publish a message to a non existing topic , it first time fails and then successfully…
Ratha
  • 9,434
  • 17
  • 85
  • 163
40
votes
4 answers

Kafka consumer fetching metadata for topics failed

I am attempting to write a Java client for a third party's Kafka and ZooKeeper servers. I am able to list and describe topics, but when I attempt to read any, a ClosedChannelException is raised. I reproduce them here with the command line client. $…
jwilner
  • 6,348
  • 6
  • 35
  • 47
38
votes
1 answer

How does Kafka store offsets for each topic?

While polling Kafka, I have subscribed to multiple topics using the subscribe() function. Now, I want to set the offset from which I want to read from each topic, without resubscribing after every seek() and poll() from a topic. Will calling seek()…
Raunaq Kochar
  • 1,024
  • 3
  • 15
  • 24
38
votes
5 answers

Kafka how to read from __consumer_offsets topic

I'm trying to find out which offsets my current High-Level consumers are working off. I use Kafka 0.8.2.1, with no "offset.storage" set in the server.properties of Kafka - which, I think, means that offsets are stored in Kafka. (I also verified that…
Marina
  • 3,894
  • 9
  • 34
  • 41
37
votes
3 answers

Kafka10.1 heartbeat.interval.ms, session.timeout.ms and max.poll.interval.ms

I am using kafka 0.10.1.1 and confused with the following 3 properties. heartbeat.interval.ms session.timeout.ms max.poll.interval.ms heartbeat.interval.ms - This was added in 0.10.1 and it will send heartbeat between polls. session.timeout.ms -…
user1578872
  • 7,808
  • 29
  • 108
  • 206
1
2
3
…
99 100