python requests chunked response

Question

I am using the Python Requests package to write a simple rest client. Here is my code -

r = requests.get(url, auth=(user, passwd), stream=True, verify=False)
print('headers: ')
pprint.pprint(r.headers)
print('status: ' + str(r.status_code))
print('text: ' + r.text)

Here is the output -

headers: 
    {'content-type': 'text/xml;charset=UTF-8',
     'date': 'Thu, 16 May 2013 03:26:06 GMT',
     'server': 'Apache-Coyote/1.1',
     'set-cookie': 'JSESSIONID=779FC39...5698; Path=/; Secure; HttpOnly',
     'transfer-encoding': 'chunked'}
status: 200

Traceback (most recent call last):
  File "C:\...\client.py", line 617, in _readinto_chunked
    chunk_left = self._read_next_chunk_size()
  File "C:\...\client.py", line 562, in _read_next_chunk_size
    return int(line, 16)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xce in position 0: invalid continuation byte

The response to that request is XML. Looks like it is chunked. Is there a special way to read chunked response? I would like to put the entire XML response in one string.

Ian Stapleton Cordasco · Accepted Answer · 2013-05-17T13:21:05.710

3

You only ever use stream=True when you're planning on iterating over the content of the response. If you are planning to print the response content immediately then stream=True will not give you any performance benefits. It will only ever defer loading the content into memory until you call r.text or r.content and then it will be loaded into memory. If you want to prevent loading the entire content into memory, check below. For the other issue, try this:

print('text:')
print(r.text)

or

print('text: ' + r.content)

If you're on 2.x, r.text is a unicode object which may not be able to be transformed to ASCII.

I'm not quite sure why chunked responses wouldn't work without stream=True, but the only way to use it properly (without downloading all of it at once like you do with r.content or r.text) is to use either iter_content or iter_lines. To collect all of the response content into one string, you can do the following:

contents = ''.join(r.iter_content(224))  # stole the number from your comment

On a related note: using the decode method on the returned string will provide highly inconsistent results. If your API allows it send the Accept-Encoding header so you can always be sure to get back data you can decode.

You're not already doing that, so I didn't suggest it, but if you're insistent on print the information, then you're going to need it, especially if it is an API to a internationally popular website.

edited May 17 '13 at 13:21

answered May 16 '13 at 22:51

Ian Stapleton Cordasco

26,944
4
67
72

sorry, but your statement about stream is not true. Requests documentation - "By default, when you make a request, the body of the response is downloaded immediately. You can override this behavior and defer downloading the response body until you access the Response.content attribute with the stream parameter" – Quest Monger May 17 '13 at 05:13
Continued - not using 'stream=true' for chunked responses results in really weird errors like the one i have shown above. i just tried it, and i got this error - "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 2: invalid start byte" i tried the other options you mentioned, and i get the same error. howvever, this works - rString = r.raw.read(224).decode() here, 224 represent the response byte length. – Quest Monger May 17 '13 at 05:13
@QuestMonger I know `stream=True` is not causing the `UnicodeDecodeError`. That's an encoding issue between the server and you (i.e., trying to print the data). Regardless, I updated my answer with a way to collect all of the response into one string. – Ian Stapleton Cordasco May 17 '13 at 13:22
the UnicodeDecodeError problem was with the restful api. it had some encoding/response issues. i am going to accept above answer as its the closest to a solution we got. – Quest Monger Sep 17 '14 at 20:57

python requests chunked response

1 Answers1