0

With the aim of getting consumer group to topic mapping, similar to that in Kafka CLI, I was trying to use the describe_config api offered by AdminClient

./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group testing-group 
GROUP           TOPIC            PARTITION  CURRENT-OFFSET    
testing-group   testing-topic-xx 0          5                 
testing-group   testing-topic-xx 1          7                 

          
LOG-END-OFFSET  LAG             CONSUMER-ID                     
5               0               console-consumer-1289
9               2               console-consumer-3456

However, while doing that I get the below exception. I'm using this example as reference.Further, the describe_config api works while describing a topic but generates below exception while describing consumer group?

KafkaError{code=INVALID_REQUEST,val=42,str="This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details."}

reference:

kadmin = AdminClient({'bootstrap.servers': 'localhost:9092, localhost:9093'})
groups = kadmin.list_groups(timeout=10)
res_list = []
res_list.append(ConfigResource(ConfigResource.Type.GROUP, "testing-group-new"))
group_cfg = kadmin.describe_configs(res_list)

Once I have the group_cfg, printing it as done here raises the exception shared above.

Tushar
  • 528
  • 4
  • 20
  • What exactly are you trying to get? Consumer groups don't have a "config", so even the Java API doesn't support such a describe config query. What resource type did you use? Are you trying to get topics in a group? I've already answered how you get lag – OneCricketeer Jun 01 '22 at 04:48
  • To get consumer group info, this example may be better to get what you want https://github.com/confluentinc/confluent-kafka-python/blob/80ea78c8e46823b68372448e49153723dfffdbf8/examples/adminapi.py#L276 – Gerard Garcia Jun 01 '22 at 05:39
  • @OneCricketeer: Trying to get the list of topics in a consumer group. Resource type I used was `GROUP`. Updated the question with the code. @GerardGarcia: Thanks, but I tried that as well. It doesn't give me list of topics linked to each consumer group name. – Tushar Jun 01 '22 at 13:58
  • It might be worth look [at the commit that added the `list_groups` method](https://github.com/confluentinc/confluent-kafka-python/commit/77681916850ec5902dd0b9f270f8274d73d81b25), then look at the [group protocol](https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-GroupMembershipAPI) wiki where you see `MemberAssignment` protocol object. And those [map into the Java API here](https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L598-L604), for example. – OneCricketeer Jun 01 '22 at 16:32
  • @OneCricketeer: thanks for sharing the link above. [The `metadata` and `assignment` attributes are in `byte` format](https://github.com/confluentinc/confluent-kafka-python/blob/77681916850ec5902dd0b9f270f8274d73d81b25/confluent_kafka/admin/__init__.py#L596). Tried `deserializing` using `codecs` and `decode` but generates `UnicodeDecodeError`. Tried passing a `deserializer` in `AdminClient` config similar to [json consumer](https://github.com/confluentinc/confluent-kafka-python/blob/80ea78c8e46823b68372448e49153723dfffdbf8/examples/json_consumer.py#L98) but that raises `Kafka Config error`. – Tushar Jun 01 '22 at 21:17
  • can the `list_groups` api in `confluent-kafka-python` return topic partition details as String similar to `list_consumer_group_offsets` in [`kafka-python`](https://stackoverflow.com/questions/61648041/find-the-topics-under-a-consumer-group-in-kafka-using-kafka-python)? – Tushar Jun 03 '22 at 01:19

0 Answers0