0

I try to call multiple pykafka consumer function using async. However, the first pykafka consumer function will block the other function from working.

The QueueConsumer lib:

import json
from pykafka import KafkaClient
import configparser

import asyncio


class QueueConsumer(object):

    def __init__(self):
        config = configparser.ConfigParser()
        config.read('config.ini')
        self.config = config

    async def test(self):
        defaultTopic = 'test'
        client = KafkaClient(hosts=self.config['kafka']['host'])
        topic = client.topics[defaultTopic.encode('utf-8')]
        consumer = topic.get_simple_consumer()
        # msg = next(consumer)
        for message in consumer:
            print(defaultTopic+' '+message.value.decode("utf-8"))

    async def coba(self):
        defaultTopic = 'coba'
        client = KafkaClient(hosts=self.config['kafka']['host'])
        topic = client.topics[defaultTopic.encode('utf-8')]
        consumer = topic.get_simple_consumer()
        # msg = next(consumer)
        for message in consumer:
            print(defaultTopic+' '+message.value.decode("utf-8"))

Then I call those function using:

import asyncio
queueConsumer = QueueConsumer()
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(
    queueConsumer.test(),
    queueConsumer.coba(),
))
loop.close()

The result will only return queue message from topic 'test' only.

Edit: I try to add another function

async def factorial(self, name, number):
    f = 1
    for i in range(2, number+1):
        print("Task %s: Compute factorial(%s)..." % (name, i))
        await asyncio.sleep(1)
        f *= i
    print("Task %s: factorial(%s) = %s" % (name, number, f))

And then called like:

    queueConsumer.test(),
queueConsumer.coba(),
queueConsumer.factorial('a',3),
queueConsumer.factorial('b',5),
queueConsumer.factorial('c',7),

Some print from factorial function is executed. But when print from either test or coba is called, then it just stop the others.

1 Answers1

0

SimpleConsumer.consume is a blocking call, so you'll need to adjust your code to periodically poll for new messages while relinquishing control between polls for other asynchronous operations to take over. One way this could be accomplished is with the use_greenlets=True kwarg on KafkaClient, relying on gevent to handle control flow between multiple asynchronous operations.

Emmett Butler
  • 5,969
  • 2
  • 29
  • 47