Streaming Failing with Requests and Flask+Celery

Question

Background: I've seen similar issues before (like here and here), but the first one never seemed to get resolved, and the second is for an older version.

I'm testing Flask's streaming responses, as part of changes we're going to make to our API. I have a Flask site set up on localhost, a and a celery queue. Both are working correctly, and tasks are showing up and being finished in the Celery queue.

The Flask site is a blueprint, setup like this:

from flask import Blueprint,  Response
from time import sleep
from celery import Celery
import logging

log = logging.getLogger(__name__)
cel = Blueprint('celerytest', __name__)
list_nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
celery = Celery('flask_test', broker='amqp://localhost:5672',
                backend='amqp://localhost:5672')


@celery.task()
def squared(number):
    result = number ** 2
    print result #Shows up inside celery so I know it's working
    sleep(result)
    rstr = str(result) + '\n'
    return rstr


@cel.route('/view', methods=['GET'])
def view_two():
    tasks = []
    for a in list_nums:
        tasks.append(squared.delay(a))

    def generate(things):
        for task in things:
            print 'doin it'
            yield task.get()

    return Response(generate(tasks),  direct_passthrough=True)

It's then attached to an app which runs on localhost:1000. To test it, I'm running a script:

def gen_lines():
    req = requests.Request(url='http://localhost:1000/testing/view',
                           method='GET').prepare()
    s = requests.Session()
    resp = s.send(req, stream=True)
    for line in resp.iter_lines():
        if line:
            yield line

print "Not blocking, honest"

for line in gen_lines():
    print line

I'm currently seeing results something like this on the Flask server:

doin it 
doin it
doin it

And etc. They take place at the correct time (e.g. 1 second, 4 seconds, 9 seconds, etc.)

When running the script, I see

Not blocking, honest

And then 196 seconds later:

1
4
9
16
etc...

I assume that once I set things up correctly, I'll see 1 (after one second), 4 (after four seconds), and etc, instead of the whole list being iterated through at the end.

Previously, my script was:

post = requests.get('http://localhost:1000/testing/view', stream=True)
for line in post.iter_lines():
    print line

But I changed it on the advice of the first answer I linked. The change didn't seem to help.

Does anyone know if I'm missing an option on either side that's causing streaming to fail?

Thanks!

ETA: I've just tested with curl (curl -G http://localhost:1000/testing/view), and the streaming is working no problem through curl. So this issue has to be with the requests module.

Do you have any middleware enabled? What WSGI setup are you using? — Martijn Pieters, Jan 21 '14 at 22:29
I'm using a virtual environment with only Celery, requests and Flask installed. This should mean no middleware, and the default WSGI setup right? ETA: requests is in the virtual environment too. — Zachary Jacobi, Jan 22 '14 at 14:08
Presumably; how are you running Flask? Using the built-in server? — Martijn Pieters, Jan 22 '14 at 14:10
app.run('localhost', port=1000) under my if __name__ == '__main__':, then calling the script from the command prompt. — Zachary Jacobi, Jan 22 '14 at 14:19
Right; that, at least, rules out the possibility of the server or middleware to interfere. — Martijn Pieters, Jan 22 '14 at 14:20
I've now tested with curl, and streaming works through it, so I'm thinking the problem lies somewhere in requests. — Zachary Jacobi, Jan 22 '14 at 14:49

Streaming Failing with Requests and Flask+Celery

0 Answers0