Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
16
votes
2 answers

No current assignment for partition occurs even after poll in Kafka

I have Java 8 application working with Apache Kafka 2.11-0.10.1.0. I need to use the seek feature to poll old messages from partitions. However I faced an exception of No current assignment for partition which is occurred every time I am trying to…
Praytic
  • 1,771
  • 4
  • 21
  • 41
16
votes
2 answers

Control enabling/disabling Kafka consumers in Spring Boot

I have configured several Kafka consumers in Spring Boot. This is what the kafka.properties looks like (only listing config for one consumer…
Hua
  • 666
  • 2
  • 9
  • 21
16
votes
2 answers

Consumer Stuck in Re-join

I've read other threads and I've gotten around the problem by using a new group ID, however I'd like to understand what could cause this. I have a topic with 16 partitions, I've set session.timeout.ms=30000, and max.poll.interval.ms=30000000. I run…
kyl
  • 475
  • 1
  • 5
  • 16
16
votes
4 answers

Kafka Consumer outputs excessive DEBUG statements to console (ecilpse)

I'm running some sample code from http://www.javaworld.com/article/3060078/big-data/big-data-messaging-with-kafka-part-1.html?page=2, and the kafkaconsumer is consuming from topic as desired, but every poll results in print (to std out) of many…
Bay Wei Heng
  • 163
  • 1
  • 1
  • 5
16
votes
1 answer

Why does a Kafka consumer take a long time to start consuming?

We start a Kafka consumer, listening on a topic which may not yet be created (topic auto creation is enabled though). Not long thereafter a producer is publishing messages on that topic. However, it takes some time for the consumer to notice this:…
Raf
  • 842
  • 1
  • 9
  • 25
16
votes
3 answers

Kafka bootstrap-servers vs zookeeper in kafka-console-consumer

I'm trying to test run a single Kafka node with 3 brokers & zookeeper. I wish to test using the console tools. I run the producer as such: kafka-console-producer --broker-list localhost:9092,localhost:9093,localhost:9094 --topic testTopic Then I…
nico
  • 2,022
  • 4
  • 23
  • 37
16
votes
7 answers

NoBrokersAvailable: NoBrokersAvailable-Kafka Error

i have already started to learn Kafka. Trying basic operations on it. I have stucked on a point which about the 'Brokers'. My kafka is running but when i want to create a partition. from kafka import TopicPartition (ERROR THERE) consumer =…
Beyhan Gul
  • 1,191
  • 1
  • 15
  • 25
16
votes
5 answers

Simple Kafka Consumer Example not working

I have a simple class to consume messages from a kafka server. The majority of codes are copied from the comments of org.apache.kafka.clients.consumer.KafkaConsumer.java. public class Demo { public static void main(String[] args) { …
David
  • 420
  • 2
  • 5
  • 18
15
votes
2 answers

Receiving Kafka Key in spring boot kafka listener

I am new in Spring Kafka. I have an microservice which sends message with a kafka key which is an user defined object. 1) First microservice sends message to Kafka with a key which is instance of MyKey object. 2) What I need to do is, to listen…
user1474111
  • 1,356
  • 3
  • 23
  • 47
15
votes
2 answers

Kafka as a message queue for long running tasks

I am wondering if there is something I am missing about my set up to facilitate long running jobs. For my purposes it is ok to have At most once message delivery, this means it is not required to think about committing offsets (or at least it is ok…
Gerard Murphy
  • 211
  • 1
  • 2
  • 6
15
votes
6 answers

How to choose the no of partitions for a kafka topic?

We have 3 zk nodes cluster and 7 brokers. Now we have to create a topic and have to create partitions for this topic. But I did not find any formula to decide that how much partitions should I create for this topic. Rate of producer is 5k…
Raj
  • 429
  • 2
  • 6
  • 26
15
votes
2 answers

Kafka Rebalancing. Duplicate processing issue

I have a consumer worker application, that internally is launching X number of threads, each thread is spawning it's KafkaCosnumer. Cosnumers have the same groupId and are subscribed to the same topics. So, that each consumer gets it fair share of…
Ihor M.
  • 2,728
  • 3
  • 44
  • 70
15
votes
7 answers

kafka-python consumer not receiving messages

I am having trouble with KafaConsumer to make it read from the beginning, or from any other explicit offset. Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning option and it hangs…
Karthik Raj
  • 241
  • 2
  • 3
  • 6
14
votes
2 answers

How to consume messages between two timestamps using Kafka Console Consumer

Is it possible to retrieve messages in Kafka console conumer for a particular timestamp range? For example kafka messags between 08:00 to 09:00 yesterday.
user11311618
14
votes
1 answer

Consuming from single kafka partition by multiple consumers

I read following in kafka docs: The way consumption is implemented in Kafka is by dividing up the partitions in the log over the consumer instances so that each instance is the exclusive consumer of a "fair share" of partitions at any point in…
MsA
  • 2,599
  • 3
  • 22
  • 47
1 2 3
…
99 100