kafka kafka-consumer-groups.sh --describe returns no output for a consumer group

Question

kafka version 1.1

--list can get the consumers group

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list --command-config config/client_security.properties
Note: This will not show information about old Zookeeper-based consumers.

spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0
console-consumer-23379
console-consumer-44846
console-consumer-75221
console-consumer-57833
yizhisec-traffic
console-consumer-91940
group1

can't get any information. This consumer group id is generated by spark.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0  --command-config config/client_security.properties

get nothing

Note: This will not show information about old Zookeeper-based consumers.

But test other consumers work.

update more detail

I use Structured Streaming kafka

and when I use zookeeper to query get error

bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0  --command-config config/client_security.properties

get error

Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).
Error: The consumer group 'spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0' does not exist.

what version of spark-streaming-kafka-* are you using?, old kafka client version (below 0.9) seems to store kafka offset on Zookeeper and not show in this. — Tuyen Luong, Apr 14 '20 at 10:27
Also, please try to copy-paste your commands/output instead of sharing a screenshot. — Giorgos Myrianthous, Apr 14 '20 at 10:44
What is the output of `bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server localhost:9092 --list --command-config config/client_security.properties ` ? Note the `--new-consumer` flag. — Giorgos Myrianthous, Apr 14 '20 at 10:56

Giorgos Myrianthous · Accepted Answer · 2020-04-14T14:28:38.447

kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --describe \
  --group your_consumer_group_name

will return no output if the consumers in the consumer group have not committed any offsets.

If the output of

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe \
  --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 \
  --command-config config/client_security.properties

and

bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --new-consumer
  --describe \
  --group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 \
  --command-config config/client_security.properties

matches and the consumer group appears in both consumer groups list output, my guess is that your consumers inside the consumer group spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 have not committed any offsets and this is why no information appears using the describe command.

Make sure that your consumers inside spark-kafka-source-fd469ba1-6ffa-43cb-8919-06c45ae56fd6--724816166-driver-0 commit their offsets succesfully (either manually or automatically).

I like that answer! Maybe also worth mentioning that Spark Structured Streaming usually does not commit offsets to Kafka but rather uses checkpoints as described [here](https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#kafka-specific-configurations). — Michael Heil, Apr 14 '20 at 12:57

kafka kafka-consumer-groups.sh --describe returns no output for a consumer group

update more detail

1 Answers1