3

I've been utilizing PycURL for my project https://pypi.python.org/pypi/tidehunter which makes use of the pycurl.WRITEFUNCTION hook to consume incoming HTTP stream in a controllable way. Use cases can be found on the project page, or the technique like this:

def _on_data_receive(self, data):
    self.buffer += data

    if data.endswith('\r\n'):  # or any valid chunk delimiter
        # Do something about self.buffer
        print self.buffer

    if True:  # some valid condition to disconnect
        return -1  # the way to stop the stream, would raise pycurl.error

I'm considering switching out of PycURL as the solution I'm using to terminate stream on-demand is rather hacky (also happy to gain better solution for that use case with PycURL). Also requests is a much more pleasant library to use.

So is there anything in requests that I can leverage to achieve the same purpose? Or maybe it's also something like a incoming stream handler that I need to explore.

Thanks in advance.

woozyking
  • 4,880
  • 1
  • 23
  • 29
  • Looks relevant (specifically the `iter_content` and `iter_lines` methods): http://www.python-requests.org/en/latest/user/advanced/#body-content-workflow – Ismail Badawi Nov 07 '13 at 20:06
  • Thanks @IsmailBadawi. I've explored http://www.python-requests.org/en/latest/user/advanced/#streaming-requests before and that was the solution I used. The decision to use `pycurl` instead was due to the write function hook's capability to `return -1` in order to terminate the stream based on custom condition. – woozyking Nov 07 '13 at 20:32
  • 1
    [Event hooks](http://docs.python-requests.org/en/latest/user/advanced/?highlight=write%20function#event-hooks) maybe? – decached Nov 14 '13 at 11:53
  • @woozyking: you can use `stream=True`, then read from `response.raw` and close that when done, perhaps. – Martijn Pieters Dec 19 '13 at 11:10

1 Answers1

1

EDIT 2016-03-22: Apologize for the selfishness.

Current Answer

The following snippet is what I applied to fully replace pycurl using python-requests. In short, simply supply with stream=True argument.

import requests

def tide_on(url):
    r = requests.get(url, stream=True, **kwargs)

    for line in r.iter_lines():
        # wish to stop after consuming just one line
        print(line)
        break

    r.close()

Previous "Answer"

The current version of requests at the time of speaking really made it easy to implement what I asked. Huge thanks to the team behind requests!

I fully replaced PycURL with requests in tidehunter since version 1.0 and it's working great!

Thanks again to all the comments above :)

woozyking
  • 4,880
  • 1
  • 23
  • 29
  • Please write how you solved it in the answer, so others can benefit too. This is not actually an answer to your question. – Hjulle Jul 26 '15 at 10:11
  • 1
    @Hjulle thanks for the advice. I've eventually come around to edit this with a more answer-like answer. Hope it helps more people than my previous selfish comment. – woozyking Mar 22 '16 at 20:59