2

I am trying to create an async Python app that subscribes to a Kafka topic and prints out the received messages via the asyncio.Task named echo_kafka_msgs_task.

At the same time, there is a second asyncio.Task named ping_task that prints out an incrementing integer every second.

However, the integer is only printed out once, and no longer prints out when echo_kafka_msgs_task has started.

Why is this happening, and how can we fix this?

import asyncio
import json
import discord
from kafka import KafkaConsumer


class DiscordBot(discord.Client):
    def __init__(self):
        super().__init__()
        self.consumer = KafkaConsumer(
            "my_topic",
            value_deserializer=lambda m: json.loads(m.decode('utf-8'))
        )
        
    async def on_ready(self):
        ping_task = asyncio.create_task(self.ping())
        echo_kafka_msgs_task = asyncio.create_task(self.echo_kafka_msgs())
        
    async def echo_kafka_msgs(self):
        while True:
            self._logger.info("Waiting for new messages...")
            for msg in self.consumer:
                print(msg.value)
            
    async def ping(self):
        i = 0
        while True:
            print("i=", i)
            i += 1
            await asyncio.sleep(1)    

if __name__ == '__main__':
    client = DiscordBot()
    client.run("my_token")
Athena Wisdom
  • 6,101
  • 9
  • 36
  • 60
  • 2
    It seems that `kafka` is a blocking (synchronous) library not suited for asynchronous projects, you could use something like [`aiokafka`](https://pypi.org/project/aiokafka/) instead, which supports async programming. – Łukasz Kwieciński Nov 08 '21 at 16:25
  • @ŁukaszKwieciński Tried `aiokafka` and now its working as expected! Thanks! – Athena Wisdom Nov 08 '21 at 16:33
  • I'm also interested to learn about how an `asyncio` event loop can get blocked by something synchronous. – Athena Wisdom Nov 08 '21 at 16:34
  • You just did learn that :-). Only code containing an "await" expression will co-operate with other tasks. Only await expressions yield control back to the event loop, which is the trigger that allows other tasks to run. – Paul Cornelius Nov 09 '21 at 04:52
  • @AthenaWisdom, hello! Did you solve the problem? I got stuck with the same problem, where `kafka-python`'s producer does not work in `multiprocessing.Process` – Ivan Feb 11 '22 at 11:21

0 Answers0