How to get the raw content of a response in requests with Python?

Question

Trying to get the raw data of the HTTP response content in requests in Python. I am interested in forwarding the response through another channel, which means that ideally the content should be as pristine as possible.

What would be a good way to do this?

score 37 · Answer 1 · answered Oct 16 '17 at 14:23

37

After requests.get(), you can use r.content to extract the raw Byte-type content.

r = requests.get('https://yourweb.com', stream=True)
r.content

answered Oct 16 '17 at 14:23

William

4,258
2
23
20

Blair · Accepted Answer · 2022-08-04T21:01:13.847

35

If you are using a requests.get call to obtain your HTTP response, you can use the raw attribute of the response. Here is the code from the requests docs. The stream=True parameter in the requests.get call is required for this to work.

>>> r = requests.get('https://github.com/timeline.json', stream=True)
>>> r.raw
<requests.packages.urllib3.response.HTTPResponse object at 0x101194810>
>>> r.raw.read(10)
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'

edited Aug 04 '22 at 21:01

answered Jun 04 '13 at 17:45

Blair

6,623
1
36
42

OK, that seems like it's promising... How would I get the whole data from that? – Juan Carlos Coto Jun 04 '13 at 18:10
9

looks like it's `r.raw.data` – Blair Jun 04 '13 at 18:37
5

This doesnt seem to work correctly, I tried `res.raw.data` and `res.raw.read(100)` but they both return empty. – Doron Cohen Aug 18 '15 at 05:48
11

@DoronCohen Did you include `stream=True` ? – farthVader Oct 20 '15 at 23:27
9

you could use use `r.raw.decode_content = True` to handle `Content-Encoding` http header. – jfs Oct 23 '15 at 12:22
@farthVader Thanks, I missed that one. – Doron Cohen Nov 05 '15 at 16:37
2

You specifically said if you have used "get"; Does that mean we cannot use this for response object generated as a result of "post"? – Sourabh Desai Jan 31 '20 at 04:44
@farthVader that hint with `stream=True` was a lifesaver! It should be a separate answer, because I had the exact same problem, and only that small change did it. – Devolus Jun 02 '22 at 08:26
@Devolus Actually the hyperlinked docs emphasize this(make sure `stream=True` before access `r.raw`). I'd like to edit this answer, but the edit queue is full. – YouJiacheng Aug 01 '22 at 04:58
I've edited the answer to emphasize the requirement of `stream=True`. – Blair Aug 04 '22 at 21:01
This response is incomplete without mentioning `r.content` as in [this other answer](https://stackoverflow.com/a/46772625/1048186) – Josiah Yoder May 02 '23 at 19:12

score 4 · Answer 3 · edited May 03 '23 at 18:28

4

To add to @blair answer, as stated in the docs:

In general, however, you should use a pattern like this to save what is being streamed to a file:

r = requests.get('https://yourweb.com', stream=True)

with open(filename, 'wb') as fd:
   for chunk in r.iter_content(chunk_size=128):
      fd.write(chunk)

Using Response.iter_content will handle a lot of what you would otherwise have to handle when using Response.raw directly. When streaming a download, the above is the preferred and recommended way to retrieve the content. Note that chunk_size can be freely adjusted to a number that may better fit your use cases.

That pattern not only has the advantages described above, but is also a good to fetch data in environments with limited memory.

edited May 03 '23 at 18:28

Blair

6,623
1
36
42

answered Aug 11 '20 at 20:29

rodrigo-silveira

12,607
11
69
123

1

Doesn't seem a complete answer. What is `r`? – robertspierre Nov 19 '21 at 17:10
1

I edited to add clarity, but had you looked at the two other answers, I was simply following their pattern of setting `r` to be the request object. – rodrigo-silveira Nov 23 '21 at 06:24
This answer worked perfectly for me. Thanks. I also thought that 'r' is intuitive (esp. given the other responses). – Mark Dec 20 '21 at 21:15

score 1 · Answer 4 · answered Jul 04 '23 at 10:26

The following is an easy way to recreate the whole HTTP response, including the HTTP header's initial Status Line:

r = requests.get('https://yourweb.com/', stream=True)
print(f"HTTP/{r.raw.version/10} {r.raw.status} {r.raw.reason}")
for k,v in r.raw.headers.items(): print(f"{k}: {v}")
print(r.text)

This may not be 100% pristine, but it should be very close. And you could use print()'s file= parameter to redirect the output to a file.

How to get the raw content of a response in requests with Python?

4 Answers4

Linked

Related