Twisted multiple concurrent or async streams

Question

I'm writing an application in python using the twisted.web framework to stream video using html 5.

The videos are being server via static.File('pathtovideo').render_GET() The problem is that only one video can be streamed at a time as it ties up the entire process.

Is there anyway to make the streaming async or non-block, whichever term would be appropriate here.

I tried using deferToThread but that still tied up the process.

This is the class Im currently using, where Movie is an ORM table and mid is just an id to an arbitrary row.

class MovieStream(Resource):
    isLeaf=True

    def __init__(self, mid):
        Resource.__init__(self)
        self.mid = mid

    def render_GET(self, request):
        movie = Movie.get(Movie.id == self.mid)
        if movie:
             defered = deferToThread(self._start_stream, path=movie.source), request=request)
             defered.addCallback(self._finish_stream, request)
             return NOT_DONE_YET
        else:
            return NoResource()
`
     def _start_stream(self, path, request):
         stream = File(path)
         return stream.render_GET(request)

     def _finish_stream(self, ret, request):
         request.finish()

The premise of this question is incorrect. Try http://sscce.org/ — Jean-Paul Calderone, Apr 05 '14 at 16:58

score 3 · Answer 1 · answered Apr 05 '14 at 21:55

The part of this code that looks like it blocks is actually the Movie.get call.

It is incorrect to call _start_stream with deferToThread because _start_stream uses Twisted APIs (File and whatever File.render_GET uses) and it is illegal to use Twisted APIs except in the reactor thread (in other words, it is illegal to use them in a function you call with deferToThread).

Fortunately you can just delete the use of deferToThread to fix that bug.

To fix the problem that Movie.get blocks you'll need to find a way to access your database asynchronously. Perhaps using deferToThread(Movie.get, Movie.id == self.mid) - if the database library that implements Movie.get is thread-safe, that is.

For what it's worth, you can also avoid the render_GET hijinx by moving your database lookup logic earlier in the resource traversal hierarchy.

For example, I imagine your URLs look something like /foo/bar/<movie id>. In this case, the resource at /foo/bar gets asked for <movie id> children. If you implement that lookup like this:

from twisted.web.resource import Resource
from twisted.web.util import DeferredResource

class MovieContainer(Resource):
    def getChild(self, movieIdentifier):
        condition = (Movie.id == movieIdentifier)
        getting = deferToThread(Movie.get, condition)
        return DeferredResource(getting)

(assuming here that Movie.get is thread-safe) then you'll essentially be done.

Resource traversal will conclude with the object constructed by DeferredResource(getting) and when that object is rendered it will take care of waiting for getting to have a result (for the Deferred to "fire", in the lingo) and of calling the right method on it, eg render_GET, to produce a response for the request.

This looks perfect! I never found deferred Resource... Thank you so much, I'll let you know how it goes — user3501657, Apr 06 '14 at 01:33

Twisted multiple concurrent or async streams

1 Answers1