9

We have an issue where it appears that a Kafka consumer is not receiving messages published to a topic. (I say appears as I have not yet got to the bottom of this, and I could be wrong.)

I am using Spring for Apache Kafka and my consumer is actually a method annotated with @KafkaListener.

This issue is intermittent and I am having trouble recreating it.

Is there a way for me to look at the Kafka broker's logs, or any other tool to help me find out the offset for my consumer? I want concrete evidence that my consumer is receiving the message or not.

vegemite4me
  • 6,621
  • 5
  • 53
  • 79
  • 1
    There's nothing in the broker's logs (i.e. log files) that includes this information. See Sönke Liebau's answer for how to find the information you are looking for. – miguno Mar 03 '17 at 09:27

4 Answers4

12

Take a look at the kafka-consumer-groups tool, which can be used to check offsets and lag of consumers (consumer has to be active at the time you run this command).

./kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --new-consumer --describe --group console-consumer-55936

GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
console-consumer-55936         test                           0          6               6               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           1          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           2          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           3          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           4          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           5          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           6          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           7          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           8          1               1               0               consumer-1_/192.168.0.83

This should allow you to track whether anything is actually being consumed or not.

Sönke Liebau
  • 1,943
  • 14
  • 23
  • This looks perfect. Is `kafka-consumer-groups` part of a more recent version of Kafka? We are using 2.11-0.9.0.1. – vegemite4me Mar 03 '17 at 16:39
  • 1
    As far as I am aware this was introduced in [0.9.0.0](https://issues.apache.org/jira/browse/KAFKA-1476), so should be available to you. – Sönke Liebau Mar 03 '17 at 16:51
  • Ah, it looks like they missed creating the `.bat` file version for this. I've created my own locally ([as someone has since added this to the project](https://github.com/apache/kafka/commits/trunk/bin/windows/kafka-consumer-groups.bat)). Thank you again. – vegemite4me Mar 03 '17 at 18:07
  • From the output: `The [new-consumer] option is deprecated and will be removed in a future major release.The new consumer is used by default if the [bootstrap-server] option is provided.` – Ilya Serbis Aug 31 '18 at 19:27
1

Besides using ./kafka-consumer-groups as described in the other answer which is correct and perfectly valid you can also use a Windows GUI application that shows you the same information about Consumer_groups and their offset/lag:

http://www.kafkatool.com/download.html

Aleja_Vigo
  • 970
  • 7
  • 12
0

Cluster Manager for Apache Kafka, previously known as Kafka Manager shows the consumer offsets as well as consumer lags:

consumer lag

However the detailed view also shows detailed offsets by partition: consumer partitions with detailed offset info

CMAK: Cluster Manager on github

It has many features allowing to manage and monitor multiple kafka clusters, we use it for many years and are quite happy.

It is under active development and supports latest version of kafka.

It is heavier than running the kafka command line, but docker makes it much easier to setup

dovka
  • 971
  • 1
  • 8
  • 20
0
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group consumer-group1
Deepak
  • 3,134
  • 2
  • 24
  • 24