Twisted async request processing

Question

How can I do async request processing in Twisted like in Node.js?

I wrote sample with Twisted, but my app still waited an answer from long operation(I emulate this with time.sleep).

Also I don't understand how can I use reactor.callLater correct.

This is my sample of twisted app.


from twisted.web import server
from twisted.web.resource import Resource
from twisted.internet import reactor
from twisted.internet.defer import Deferred
import time

class Hello(Resource):
    def getChild(self, name, request):
        if name == '':
            return self
        print name
        return Resource.getChild(self, name, request)

    def render_GET(self, req):
        d = Deferred()
        reactor.callLater(2, d.callback, None)
        d.addCallback(lambda _: self.say_hi(req))
        return server.NOT_DONE_YET

    def say_hi(self, req):
        req.setHeader("content-type", "text/html")
        time.sleep(5)
        req.write("hello!")
        req.finish()

class Hello2(Resource):
    isLeaf = True
    def render_GET(self, req):
        req.setHeader("content-type", "text/html")
        return "hello2!"

root = Hello()
root.putChild("2", Hello2())
reactor.listenTCP(8080, server.Site(root))
reactor.run()

Edit: Now question is how to write a sync code? Please example.

time.sleep() is blocking call. If you have some heavy processing - try DeferToThread(). — Pill, Nov 16 '11 at 09:36
But there are a lot of blocking libs in python. What is recommended usage of such libs? — Stan, Nov 16 '11 at 10:08
Yep - sad but true. As I mentioned above - you can try to make it with DeferToThread, or search for some async analogs. — Pill, Nov 16 '11 at 11:24
What does "like in Node.js" mean? Blocking is blocking, Node.js doesn't magically prevent it. — Jean-Paul Calderone, Nov 16 '11 at 12:12
Hm, I agree. Plus to Node.js is great number of async libraries. — Stan, Nov 16 '11 at 18:50
[An Introduction to Asynchronous Programming and Twisted](http://krondo.com/?page_id=1327) — jfs, Nov 17 '11 at 06:13
Here's an [example](https://github.com/zed/txfib) that demonstrates how you could handle blocking CPU intensive code. — jfs, Nov 17 '11 at 06:26

score 4 · Answer 1 · answered Nov 16 '11 at 21:06

You're already doing it... sort of.

Your problem here is that time.sleep() is a blocking call, and will therefore make your whole server stop.

If you're using that as a stand-in for something that does network I/O (like urllib), the best option is usually to do the I/O with Twisted (like twisted.web.client.getPage) rather than to try to do something to the blocking code. Twisted has lots of client libraries. These libraries will generally give you a Deferred, which you're already handling.

If you're using it as a stand-in for actually waiting, then you can create a Deferred which waits with deferLater.

If you're using it as a stand-in for something CPU intensive that doesn't grab the GIL (like PIL image encoding, or a native XML parser), or an existing native / proprietary I/O layer (like Oracle or MSSQL database bindings) that you'd prefer not to rewrite to use Twisted properly, you can invoke it with deferToThread.

However you get your Deferred, you're almost set up to handle it. You just need to adjust say_hi to return one:

def say_hi(self, req):
    req.setHeader("content-type", "text/html")
    d = getADeferredSomehow()
    def done(result):
        req.write("hello!")
        req.finish()
    return d.addBoth(done)

Thanks for the answer. time.sleep is just a stub for blocking call. I tried to emulate call of blocking lib(like lots of libs in python) and to learn how can I write non-blocking app. — Stan, Nov 17 '11 at 06:39

Twisted async request processing

1 Answers1