Obtaining data from blocking source in Gevent loop callback

Question

I'm trying to use Pycurl with Gevent in order to perform HTTP uploads. For this I'm relying on the geventcurl.py module, which alters libcurl's Multi API to use Gevent's event loop.

The problem resides in the READFUNCTION callback. This callback is executed in the HUB context, and so we cannot wait(), but this callback must return the data to upload and in my case this data comes from a blocking source.

Here is a snippet of code that demonstrates the problem:

#!/usr/bin/env python
from gevent import monkey; monkey.patch_all()
import gevent
from gevent.queue import Queue
import pycurl
from geventcurl import Curl

URL = 'http://localhost:8000/'

class QueueReader:
    def __init__(self, q):
        self.q = q
    def read_callback(self, size):
        return self.q.get(timeout=10)

dataq = Queue(10)
c = Curl()
c.setopt(pycurl.URL, URL)
c.setopt(pycurl.UPLOAD, 1)
c.setopt(pycurl.READFUNCTION, QueueReader(dataq).read_callback)

# Start transfer
g = gevent.spawn(c.perform)
for i in xrange(10):
    dataq.put(str(i))
    gevent.sleep(1)
g.join()
c.close()

To run the snippet you just need to have something listening on localhost:8000, nc -l 8000 will do. What happens is that, since read_callback() is executed in HUB context, it will not wait, if the queue is empty it will immediately raise an Empy exception. Using an AsyncResult also does not help since we have to wait() for the result.

Is there any way of fetching data from a possibly blocking source in the event loop callback?

kkurian · Answer 1 · 2012-10-03T18:01:00.407

If you get rid of the timeout=10, it will wait for data from the blocking source. (See: http://www.gevent.org/gevent.queue.html, default behavior.)

If read_callback()'s blocking will interfere with the operation of the for i in xrange(10) loop, then put that loop into its own greenlet via, e.g., gevent.spawn().

Also note that by default put() will block if the queue is full. Consider put_nowait(), put(timeout=...), or put(block=False) to alter the default behavior.

Obtaining data from blocking source in Gevent loop callback

1 Answers1