Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
26
votes
1 answer

Is it possible to create a kafka topic with dynamic partition count?

I am using kafka to stream the events of page visits by the website users to an analytics service. Each event will contain the following details for the consumer: user id IP address of the user I need very high throughput, so I decided to…
vivek_jonam
  • 3,237
  • 8
  • 32
  • 44
25
votes
2 answers

Is a web frontend producing directly to a Kafka broker a viable idea?

I have just started learning Kafka. So trying to build a social media web application. I am fairly clear on how to use Kafka for my backend ( communicating from backend to databases and other services). However, I am not sure how should frontend…
user3271166
  • 573
  • 1
  • 6
  • 17
24
votes
8 answers

kafka failed authentication due to: SSL handshake failed

I have to add encryption and authentication with SSL in kafka. This is what I have done: Generate certificate for each broker kafka: keytool -keystore server.keystore.jks -alias localhost -validity 365 -genkey Create CA. The generated CA is a…
xRobot
  • 25,579
  • 69
  • 184
  • 304
23
votes
2 answers

When to use ConcurrentKafkaListenerContainerFactory?

I am new to kafka and i went through the documentation but I couldn't understand anything. Can someone please explain when to use the ConcurrentKafkaListenerContainerFactory class? I have used the Kafkaconsumer class but I see…
Rahul Gupta
  • 1,079
  • 2
  • 15
  • 27
23
votes
1 answer

How does Kafka Consumer Consume from Multiple assigned Partition

tl;dr; I am trying to understand how a single consumer that is assigned multiple partitions handles consuming records for reach partition. For example: Completely processes a single partition before moving to the next. Process a chunk of…
DVS
  • 783
  • 7
  • 25
23
votes
4 answers

Kafka-consumer. commitSync vs commitAsync

The quote from https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html#callout_kafka_consumers__reading_data_from_kafka_CO2-1 The drawback is that while commitSync() will retry the commit until it either…
gstackoverflow
  • 36,709
  • 117
  • 359
  • 710
23
votes
7 answers

Kafka Consumer Marking the coordinator 2147483647 dead

I am using Kafka Server 0.9 with consumer kafka-client version 0.9 and kafka-producer 0.8.2. Every thing is working great except i am getting lot of info that the coordinator is dead on the consumer 2016-02-25 19:30:45.046 INFO 10263 --- [ …
Shahbour
  • 1,323
  • 1
  • 16
  • 42
22
votes
2 answers

KafkaConsumer Java API subscribe() vs assign()

I am new with Kafka Java API and I am working on consuming records from a particular Kafka topic. I understand that I can use method subscribe() to start polling records from the topic. Kafka also provides method assign() if I want to start polling…
Karan Khanna
  • 1,947
  • 3
  • 21
  • 49
22
votes
3 answers

KafkaConsumer 0.10 Java API error message: No current assignment for partition

I am using KafkaConsumer 0.10 Java api. I want to consume from a specific partition and specific offset. I looked up and found that there is a seek method but its throwing an exception. Anyone had a similar use case or solution…
colossal
  • 529
  • 1
  • 5
  • 22
22
votes
2 answers

Kafka Consumer Rebalancing Algorithm

Can someone please tell me what the rebalancing algorithm is for Kafka consumers? I would like to understand how partition count and consumer threads affect this. Thank you,
NSA
  • 5,689
  • 8
  • 37
  • 48
21
votes
5 answers

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

I am manually starting Zookeeper, then Kafka server and finally the Kafka-Rest server with their respective properties file. Next, I am deploying my Spring Boot application on tomcat In the Tomcat log trace, I am getting the Error…
somnathchakrabarti
  • 3,026
  • 10
  • 69
  • 92
21
votes
3 answers

kafka 0.11 reset offset for consumer group by --to-datetime

I want to reset the offset of kafka consumer group by timestamp. But when I am using following command: ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --to-datetime 2017-11-1907:52:43:00:000 --group --topic…
Dhruv
  • 10,291
  • 18
  • 77
  • 126
21
votes
2 answers

InstanceAlreadyExistsException coming from kafka consumer

I am working with Kafka and trying to setup consumer group by follwing this article. The only difference is I have created my own abstract class, handler to make design simpler. Below is my abstract class: public abstract class Consumer implements…
user1950349
  • 4,738
  • 19
  • 67
  • 119
21
votes
3 answers

Limit Kafka batches size when using Spark Streaming

Is it possible to limit the size of the batches returned by the Kafka consumer for Spark Streaming? I am asking because the first batch I get has hundred of millions of records and it takes ages to process and checkpoint them.
Samy Dindane
  • 17,900
  • 3
  • 40
  • 50
21
votes
3 answers

How to make kafka consumer to read from last consumed offset but not from beginning

I am new to kafka and trying to understand if there is a way to read messages from last consumed offset, but not from beginning. I am writing an example case, so that my intention will not get deviate. Eg: 1) I produced 5 messages at 7:00 PM and…
Srini
  • 3,334
  • 6
  • 29
  • 64
1 2 3
…
99 100