I have deployed Kafka and python consumers on separate servers. I am using GCP VM instances for deployment.
Python scripts run in a loop and checks with kafka if it has a job for it. If it finds a job, it starts the operation otherwise it keeps checking with Kafka.
Scripts works just fine when entries are made in kafka on a regular basis, but exits when no entry is added to kafka queue for a certain amount of time. Thus when I push something into the kafka queue the python script cannot detect it because it has already exited.
Below is the code running on the python to keep checking the kafka for new jobs:
consumer = KafkaConsumer(bootstrap_servers=['ip-address'], group_id='grp1',
value_deserializer=lambda m: json.loads(m.decode('utf8')),
partition_assignment_strategy=[RoundRobinPartitionAssignor])
listener = Listener(consumer1)
consumer1.subscribe(['topic'], listener=listener)
while True:
messages = consumer1.poll(100, max_records=1)
for tp, msg1 in messages.items():
# Check partition(s)
print(tp)
print(msg1[0])
print("consumer1")
print(tp.partition)
print(msg1[0].value)
listener.add_offset(tp.topic, tp.partition, msg1[0].offset)
main() # does some work
print("done")
Can someone suggest what the problem is?