list_consumer_group_offsets is not returning all the partitions and their offsets

Question

I am using kafka-python to get the last offset being consumed by a consumer:

def get_consumer_group_offset(group):

    topic = ''

    g = create_admin()
    returnlist = []
    counter = 0
    
    end_offsets = g.list_consumer_group_offsets(group)
    print(end_offsets)
    offsets = list(end_offsets.values())

    offsets_u = []
    for n in offsets:
        offsets_u.append(n.offset)
    print("offsets_u")
    print(offsets_u)

    for f in end_offsets:
        topic = f.topic
        value = []
        value.append(f.partition)
        value.append(offsets_u[counter])
        counter = counter + 1
        returnlist.append(value)
    print(returnlist)
    return topic, returnlist

However I am getting the following output: (element #1 represent the partition, as you can see some partitions are being skipped for some reason)

[[1, 301], [7, 297], [10, 257], [12, 776], [14, 569], [16, 256], [20, 5], [22, 262], [28, 247]]

You're printing multiple things here. What is the output shown? — OneCricketeer, Mar 07 '22 at 21:23
I understood that part, but like `print(end_offsets)`, which you're looping over to create the returned list. Clearly something is missing in the first list — OneCricketeer, Mar 08 '22 at 13:30

score 0 · Answer 1 · edited Sep 03 '23 at 03:35

0

I had the same problem. It turns out that this is intended behavior.

Citation from description for the function list_consumer_group_offsets:

Partitions that are not specified and for which the group_id does not have a recorded offset are omitted.

See also https://github.com/dpkp/kafka-python/issues/2307

edited Sep 03 '23 at 03:35

Bihag Kashikar

1,009
1
2
13

answered Aug 31 '23 at 20:14

rafkor

1
2

list_consumer_group_offsets is not returning all the partitions and their offsets

1 Answers1