Asynchronous processing of streaming HTTP with requests / requests_futures

Question

I understand how to stream data over HTTP with requests:

import requests

r = requests.get(url, stream=True)
for line in r.iter_lines():
    print(line)

I understand how to process a request asynchronously:

from requests_futures.sessions import FuturesSession

session = FuturesSession()
def bg_cb(sess, resp):
    # parse the json storing the result on the response object
    resp.data = resp.json()

future = session.get(url, background_callback=bg_cb)
# do some other stuff, send some more requests while this one works
response = future.result()

But I don't understand how to combine the two, that is, how to get callback to my designated function whenever the next line is available from the r.iter_lines():

session = FuturesSession()
def bg_cb(sess, resp):
    # parse the json storing the result on the response object
    resp.data = resp.json()

future = session.get(url, background_callback=bg_cb, stream=True)

# do other stuff here

The code snippet calls my bg_cb function just once, before any data is actually available. Instead, I want bg_cb to be called whenever a new line is available.

@JoranBeasley I edited my last code snippet to clarify my problem. I really can't figure out a way to get a call back from a the `get` request whenever new data is available - rather than just once. — max, May 17 '16 at 18:31

score 0 · Answer 1 · answered Sep 19 '18 at 10:37

You could use pycurl, like it says in the docs:

One example could be:

import pyrcurl

STREAM_URL = "whatever/url"

def on_receive (data):
    print(data)

'''We instantiate the Curl object and setoptions so that it gets executed. 
We use the on_receive function so that the stream of data is printed as it arrives the object, which is very cool! 

The perform() operation keeps executing as it is a real time stream of data, not executing the subsequent code UNTIL we set a TIMEOUT option or we sys.exit() it with CTRL+C
'''

conn = pycurl.Curl()
conn.setopt(pycurl.URL, STREAM_URL)
conn.setopt(pycurl.WRITEFUNCTION, on_receive)
# conn.setopt(pycurl.TIMEOUT, 15)
conn.perform()
conn.close()

The pycurl.WRITEFUNCTION executes the on_receive function whenever a new data is streamed.

The pyrcul.TIMEOUT is used to specify the seconds that perform() method will be executed.

Note: the resulting object will be class <bytes>, so you will need to make the appropiate processing in the application.

Asynchronous processing of streaming HTTP with requests / requests_futures

1 Answers1