streaming views with django and kombu

Question

i have two queues in kombu; one to submit requests (to do something) and another that spits out via pub/sub the incremental status of said request. therefore in my process it will publish to the request queue and consume on the response queue. as the task may take some time, i want to provide the user with feedback as to what is happening in the backend; it all works on the command line, as my kombu consume callback allows me to, say, add a logging.info() statement to spit back information to my user:

def callback( msg, env ):
    logging.info( str(msg) )

consumer.register_callback( callback )
consumer.consume()
while continue_consuming:
    connection.drain_events()

however, i now want to be able to provide the same functionality in django. i understand that i can create a generator function as the input to a HttpResponse object:

def view( reqeust ):
    HttpResponse( gen() )

def gen():
    yield 'streaming... '

but i cannot conceptualize how i could implement the message callback of the kombu queue into a generator to provide this... any ideas?

i want to avoid having to use a database layer to store the progress/results if possible.

score 0 · Accepted Answer · answered Jun 13 '11 at 18:01

in the end i decided to restructure the code a bit; as i had a wrapper around a kombu queue to make the interface more multiprocess.Queue like, i created a generator for my get() method.

def get( self, until=None ):
    if until == None:
        until = self.end_marker
    for c in count():
        m = self.consumer.queues[0].get( True )
        if not m == None:
            if m.payload == until:
                raise StopIteration
            yield m.payload

this appears to work fine - but not all that clean as i have a need to know what self.end_marker or util is, and also might want to iterate through all the consumer queues (but my class is queue per object anyway, so that's not too bad)

then all i do in my view is:

 def view( response ):
     q = Queue()
     return HttpResponse( q.get() )

there's numerous posts about various middleware getting in the way; i just don't bother using them and it seems to work fine.

streaming views with django and kombu

1 Answers1