Python Requests/urllib — monitoring bandwidth usage

Question

I want to log the total bytes downloaded and uploaded by my Python script.

total_downloaded_bytes = 0
def bandwidth_hook(r, *args, **kwargs):
    global total_downloaded_bytes
    total_downloaded_bytes += len(r.content)
req = requests.session()
req.hooks = {'response': bandwidth_hook}

The above code doesn't take into account HTTP compression (if I'm right) and the size of headers.

Is there a way to count total uploaded and downloaded bytes from a requests.session? If not, what about a script-wide count?

score 9 · Answer 1 · edited Oct 19 '15 at 16:43

9

You can access the r.request object to calculate outgoing bytes, and you can determine incoming bytes (compressed or not) by looking at the content-length header for the incoming request. This should suffice for 99% of all requests you normally would make.

Calculating the byte size of headers is easy enough; just add up key and value lenghts, add 4 bytes for the colon and whitespace, plus 2 more for the blank line:

 def header_size(headers):
     return sum(len(key) + len(value) + 4 for key, value in headers.items()) + 2

There is also the initial line; that's {method} {path_url} HTTP/1.1{CRLF} for requests, and HTTP/1.x {status_code} {reason}{CRLF} for the response. Those lengths are all also available to you.

Total size then is:

 request_line_size = len(r.request.method) + len(r.request.path_url) + 12
 request_size = request_line_size + header_size(r.request.headers) + int(r.request.headers.get('content-length', 0))
 response_line_size = len(r.response.reason) + 15
 response_size = response_line_size + header_size(r.headers) + int(r.headers.get('content-length', 0))
 total_size = request_size + response_size

edited Oct 19 '15 at 16:43

Elmo

6,409
16
72
140

answered Oct 19 '15 at 14:33

Martijn Pieters

1,048,767
296
4,058
3,343

Wow, that looks way complex for something simple. Thanks for the answer! – Elmo Oct 19 '15 at 16:03
1

@Elmo: the respone object is a high-level modelling of HTTP information, and it was never a use case to fully reconstruct the underlying HTTP protocol bytes. – Martijn Pieters Oct 19 '15 at 16:04
Can you not hook somewhere deeper? The actual tcp stream or somewhere? – Elmo Oct 19 '15 at 16:07
@Elmo: you'd have to patch the packaged `urllib3` library that comes with `requests`, there are no ootb hooks for this. – Martijn Pieters Oct 19 '15 at 16:43

Python Requests/urllib — monitoring bandwidth usage

1 Answers1

Linked