kafka-python: Can a consumer object join a consumer group without subscribing to any topics?

Question

I have a topics_subscriber.py that continuously checks for any new topics and subscribes to them.

def _subscribe_topics(consumer: KafkaConsumer) -> None:
    topics_to_subscribe = {
        topic
        for topic in consumer.topics()
        if topic.startswith("topic-")
    }

    subscribed_topics = consumer.subscription()
    print(subscribed_topics)

    new_topics = (
        topics_to_subscribe - subscribed_topics
        if subscribed_topics
        else topics_to_subscribe
    )

    if new_topics:
        print("new topics detected:")
        for topic in new_topics:
            print(topic)

        print("\nsubscribing to old+new topics\n")
        consumer.subscribe(topics=list(topics_to_subscribe)) // subscribe() is not incremental, hence subscribing to old+new topics 

        print(consumer.subscription())
    else:
        print("\nno new topics detected: exiting normally\n")


def main() -> None:
    consumer = kafka.KafkaConsumer(
                   client_id="client_1",
                   group_id="my_group",
                   bootstrap_servers=['localhost:9092']
               )

    while True:
        _subscribe_topics(consumer)

        print("\nsleeping 10 sec\n")
        time.sleep(10)

Now, in another script kafka_extractor.py, I want to create a new consumer and join the my_group consumer group and start consuming messages from the topics that are subscribed by the group. i.e without specifically subscribing to topics for this new consumer.

def main() -> None:
    consumer = kafka.KafkaConsumer(
                   client_id="client_2",
                   group_id="my_group",
                   bootstrap_servers=['localhost:9092']
               )
    print("created consumer")
    print(consumer.subscription())

    for msg in consumer:
        print(msg.topic)

Two things to note in the output of kafka_extractor.py:

print(consumer.subscription()) outputs as None
for msg in consumer: -> is stuck and does not move forward nor exits the program.

Any directions would be appreciated here.

score 0 · Answer 1 · answered May 04 '21 at 12:39

Joining a consumer to a group does not gain the group's existing topics because groups can contain multiple topics, and consumer instances decide which topics to consume by subscribing

Your loop is stuck because it's defaulted to read from the end of the subscribed topics, but there are none

If you wanted to consume a list of topics that's refreshed periodically, use a regex subscription

kafka-python: Can a consumer object join a consumer group without subscribing to any topics?

1 Answers1