Stack Overflow
Stack Overflow

Questions tagged [kafka-consumer-api]

Use for questions related to the Apache Kafka consumer API

Use for questions related to the Apache Kafka consumer API. Any question related to consuming from the Kafka topics. Introspecting offsets. Consumers failing and restarting reliably should use this tag.

4238 questions
37
votes
2 answers

Kafka and firewall rules

We have a fairly strict network segmentation policy. I am using a cloud foundry instance to deploy an app to. The firewall rules have been set up to reach the kafka cluster from within the cloud foundry instance. I believe that the firewall rules…
George Smith
  • 1,027
  • 4
  • 11
  • 23
36
votes
4 answers

Docker Kafka w/ Python consumer

I am using dockerized Kafka and written one Kafka consumer program. It works perfectly when I run Kafka in docker and application at my local machine. But when I configured the local application in docker I am facing issues. The issue may be due to…
Pankaj Saboo
  • 1,125
  • 1
  • 8
  • 27
36
votes
8 answers

How to get latest offset for a partition for a kafka topic?

I am using the Python high level consumer for Kafka and want to know the latest offsets for each partition of a topic. However I cannot get it to work. from kafka import TopicPartition from kafka.consumer import KafkaConsumer con =…
Saket
  • 3,079
  • 3
  • 29
  • 48
35
votes
1 answer

Kafka console consumer ERROR "Offset commit failed on partition"

I am using a kafka-console-consumer to probe a kafka topic. Intermittently, I am getting this error message, followed by 2 warnings: [2018-05-01 18:14:38,888] ERROR [Consumer clientId=consumer-1, groupId=console-consumer-56648] Offset commit failed…
Yeming Huang
  • 518
  • 1
  • 4
  • 11
33
votes
2 answers

Kafka consumer offset max value?

I was googling and reading Kafka documentation but I couldn't find out the max value of a consumer offset and whether there is offset wraparound after max value. I understand offset is an Int64 value so max value is 0xFFFFFFFFFFFFFFFF. If there is…
rodolk
  • 5,606
  • 3
  • 28
  • 34
33
votes
4 answers

Get last message from kafka consumer console script

We can get every messages from Kafka by doing: bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning Is there a way to get only the last message ? EDIT: If you just want to monitor some messsages (--max-messages…
Paul Leclercq
  • 989
  • 2
  • 15
  • 26
32
votes
4 answers

Kafka producer TimeoutException: Expiring 1 record(s)

I am using Kafka with Spring-boot: Kafka Producer class: @Service public class MyKafkaProducer { @Autowired private KafkaTemplate kafkaTemplate; private static Logger LOGGER =…
Prakash P
  • 3,582
  • 4
  • 38
  • 66
31
votes
4 answers

Does Kafka support request response messaging

I am investigating Kafka 9 as a hobby project and completed a few "Hello World" type examples. I have got to thinking about Real World Kafka applications based on request response messaging in general and more specifically how to link a Kafka…
Hector
  • 4,016
  • 21
  • 112
  • 211
30
votes
2 answers

How to shift back the offset of a topic within a stable Kafka consumer group?

I try to shift back the offset of topic within consumer group using following command: bin/kafka-consumer-groups.sh --bootstrap-server loclahost:9092 --group xxx-0 --topic schedule-changed --reset-offsets --shift-by -2 --execute (Yes, I'm using…
Karol Król
  • 3,320
  • 1
  • 34
  • 37
30
votes
4 answers

How does kafka consumer auto commit work?

I am reading this one: Automatic Commit The easiest way to commit offsets is to allow the consumer to do it for you. If you configure enable.auto.commit=true, then every five seconds the consumer will commit the largest offset your client…
gstackoverflow
  • 36,709
  • 117
  • 359
  • 710
30
votes
7 answers

Counting Number of messages stored in a kafka topic

I'm using 0.9.0.0 version of Kafka and I want to count the number of messages in a topic without using the admin script kafka-console-consumer.sh. I have tried all the commands in the answer Java, How to get number of messages in a topic in apache…
jack AKA karthik
  • 885
  • 3
  • 15
  • 30
30
votes
5 answers

Kafka - Delayed Queue implementation using high level consumer

Want to implement a delayed consumer using the high level consumer api main idea: produce messages by key (each msg contains creation timestamp) this makes sure that each partition has ordered messages by produced time. auto.commit.enable=false…
Nimrod007
  • 9,825
  • 8
  • 48
  • 71
29
votes
3 answers

Not clear about the meaning of auto.offset.reset and enable.auto.commit in Kafka

I am new to Kafka,and I don't really understand the meaning of Kafka configuration, can anyone explain in a more understandable way to me? Here is my code: val kafkaParams = Map[String, Object]( "bootstrap.servers" -> "master:9092,slave1:9092", …
Gpwner
  • 559
  • 1
  • 5
  • 14
29
votes
5 answers

How to set group name when consuming messages in kafka using command line?

Any idea how to set group name when consuming messages in kafka using command line. I tried with the following command : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic nil_RF2_P2 --from-beginning --config group.id=test1 'config' is…
Nilotpal
  • 3,237
  • 4
  • 34
  • 56
27
votes
1 answer

Need clarification about Kafka auto commit and auto.commit.interval.ms

The document https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html says that "Note that with auto-commit enabled, a call to poll will always commit the last offset returned by the previous poll. It doesn’t know…
Deeps
  • 1,879
  • 4
  • 18
  • 18
1 2
3
…
99 100