Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
20
votes
3 answers

Difference between heartbeat.interval.ms and session.timeout.ms in Kafka consumer config

I'm currently running kafka 0.10.0.1 and the corresponding docs for the two values in question are as follows: heartbeat.interval.ms - The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities.…
novon
  • 973
  • 3
  • 15
  • 30
20
votes
3 answers

Consumer.poll() returns new records even without committing offsets?

If I have a enable.auto.commit=false and I call consumer.poll() without calling consumer.commitAsync() after, why does consumer.poll() return new records the next time it's called? Since I did not commit my offset, I would expect poll() would…
Glide
  • 20,235
  • 26
  • 86
  • 135
20
votes
2 answers

Kafka consumer offsets out of range with no configured reset policy for partitions

Getting below exception while starting Kafka consumer. org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions{test-0=29898318} Kafka version: 9.0.0 Java 7
basit raza
  • 661
  • 2
  • 6
  • 18
19
votes
1 answer

Why do Kafka consumers output INVALID_FETCH_SESSION_EPOCH after updating to 1.1?

We recently updated our Kafka brokers and clients to 1.1.1. Since the upgrade we periodically see INFO log entries such as INFO Jun 08 08:30:20.335 61161458 [KafkaRecordConsumer-0] org.apache.kafka.clients.FetchSessionHandler [Consumer…
Mark
  • 28,783
  • 8
  • 63
  • 92
19
votes
3 answers

How to handle kafka publishing failure in robust way

I'm using Kafka and we have a use case to build a fault tolerant system where not even a single message should be missed. So here's the problem: If publishing to Kafka fails due to any reason (ZooKeeper down, Kafka broker down etc) how can we…
User5817351
  • 989
  • 2
  • 16
  • 36
19
votes
4 answers

Kafka Consumer - Poll behaviour

I'm facing some serious problems trying to implement a solution for my needs, regarding KafkaConsumer (>=0.9). Let's imagine I have a function that has to read just n messages from a kafka topic. For example: getMsgs(5) --> gets next 5 kafka…
aran
  • 10,978
  • 5
  • 39
  • 69
18
votes
5 answers

Kafka Consumer: No entry found for connection

I am trying to check the kafka consumer by consuming the data from a topic on a remote Kafka cluster. I am getting the following error when I use the kafka-console-consumer.sh: ERROR Error processing message, terminating consumer process: …
Nooshin
  • 943
  • 1
  • 9
  • 24
18
votes
7 answers

Leader brokers without a matching listener error in kafka

What is the meaning of this kafka error ? [2018-08-22 11:40:49,429] WARN [Consumer clientId=consumer-1, groupId=console-consumer-62114] 1 partitions have leader brokers without a matching listener, including [topicname-0] …
franck
  • 2,995
  • 3
  • 17
  • 28
18
votes
4 answers

How to use Spring Kafka's Acknowledgement.acknowledge() method for manual commit

I am using Spring Kafka first time and I am not able to use Acknowledgement.acknowledge() method for manual commit in my consumer code as mentioned here https://docs.spring.io/spring-kafka/reference/html/_reference.html#committing-offsets. Mine is…
kumarhimanshu449
  • 799
  • 3
  • 7
  • 19
18
votes
1 answer

How Kafka distributes the topic partitions among the brokers

I have 3 Kafka brokers in 3 different VMs, with one additionally running a Zookeeper. I now create a topic with 8 partitions. The Producer pushes messages to these group of brokers on the created "topic". How does the Kafka distribute a topic and…
vvra
  • 2,832
  • 5
  • 38
  • 82
17
votes
1 answer

What is a listener container in Spring for Apache Kafka?

I understood that to make a method to be the target of Kafka message listener, I have to mark this method with the @KafkaListener annotation. This annotation lets specify by containerFactory element the KafkaListenerContainerFactory. Below there are…
Vin
  • 701
  • 1
  • 9
  • 30
17
votes
3 answers

Consume the same message again if processing of the message fails

I am using Confluent.Kafka .NET client version 1.3.0. I am following the docs: var consumerConfig = new ConsumerConfig { BootstrapServers = "server1, server2", AutoOffsetReset = AutoOffsetReset.Earliest, EnableAutoCommit = true, …
havij
  • 1,030
  • 14
  • 29
17
votes
1 answer

Delayed message consumption in Kafka

How can I produce/consume delayed messages with Apache Kafka? Seems like standard Kafka (and Java kafka-client) functionality doesn't have this feature. I know that I could implement it myself with standard wait/notify mechanism, but it doesn't seem…
Everv0id
  • 1,862
  • 3
  • 25
  • 47
17
votes
1 answer

Kafka consumer exception and offset commits

I've been trying to do some POC work for Spring Kafka. Specifically, I wanted to experiment with what are the best practices in terms of dealing with errors while consuming messages within Kafka. I am wondering if anyone is able to help…
yfl
  • 347
  • 2
  • 3
  • 10
17
votes
2 answers

How to use Consumer API of Kafka 0.8.2?

I'm getting start with the latest Kafka document http://kafka.apache.org/documentation.html. But I meet some problem when I try to use the new Consumer API. I've done the job with following steps: 1. Add a new dependency …
Yohn
  • 580
  • 5
  • 19
1 2 3
…
99 100