Kafka Python, How to track consumer started in a different process

Question

I am fairly new to Python and just getting started with Kafka, so pardon my terminologies if I'm wrong somewhere.

So I have a Django based web application, where I am sending a json messages through Kafka Producer within the same process. However while creating a topic pragmatically, I am also starting(subscribing) a new consumer in a separate Process for that particular topic.

#Consumer code snippet

 if topic_name is not None :
        #Create topic
        create_kafka_topic_instance(topic_name)
        #Initialize a consumer and subscribe to topic
        Process(target=init_kafka_consumer_instance, args=(topic_name))

def forgiving_json_deserializer(v):
    if v is None :
        return
    try:
        return json.loads(v.decode('utf-8'))
    except json.decoder.JSONDecodeError:
        import traceback
        print(traceback.format_exc())
        return None

def init_kafka_consumer_instance(topic, group_id=None):
    try:
        if topic is None:
            raise Exception("Invalid argument topic")
        comsumer = None
        comsumer = KafkaConsumer(topic, bootstrap_servers=[KAFKA_BROKER_URL], auto_offset_reset="earliest",
           urn comsumer
    except Exception as e:
        import traceback
        print(traceback.format_exc())
    return Noneurn comsumer
    except Exception as e:
        import traceback
        print(traceback.format_exc())
    return None

Producer Code Snippet

# assuming obj is a model instance
        serialized_obj = serializers.serialize('json', [ order, ])
        #send_message(topic_name,order)
        producer = KafkaProducer(bootstrap_servers=[KAFKA_BROKER_URL], value_serializer=lambda x: json.dumps(x).encode('utf-8'))
        x = producer.send("test", serialized_obj)
        producer.flush()

Now i have some queries, so if somehow my Django application (server) is restarted, will I still have the consumer listening to that topic.

Also I have some print statements in the consumer which I am unable to see in my server console.

However writing the same code snippet(initialising a consumer) in a python shell, I can see the messages in print statements there, meaning my Producer is working fine.

score 0 · Answer 1 · answered Apr 08 '19 at 07:28

Kafka Server is not depending on your Django application (server). But your Consumer is yes.

So your topic still alive in Kafka server (if kafka server die, it 's the other story), but your Consumer is restarted with your Application.

So if you want your consumer work well, make it a Worker that work parallel with your app and will not be restarted when your application is down

Kafka Python, How to track consumer started in a different process

1 Answers1