0

I am making a get request using Python's requests library and streaming the contents into a file. I'd like to add a progress bar by setting the max value of the progress bar to resp.headers['content-length'] and iterating through the response using chunksize=1 byte (response.iter_content(chunk_size=1)). However, the content encoding of the response is gzip so the content-length refers to the compressed size.

using the following code:

resp = requests.get(...., stream=True)
count = 0
for x in resp.iter_content(chunk_size=1):
  count += 1

i get: count=64 and resp.headers['content-length'] = 127.

What should I use as the max value of the progress bar?

Jeff Tsui
  • 1,266
  • 12
  • 20

1 Answers1

0

I would use Kenneth's clint and do it like this. Or what you should do is set your progress bar's implementation to update starting from 127.

from clint.textui import progress
import requests


resp = requests.get(url, stream=True)
total_length = int(resp.headers.get('content-length'))

for chunk in progress.bar(resp.iter_content(chunk_size=1), expected_size=(total_length / 1) + 1):
    pass
Anton Antonov
  • 1,217
  • 2
  • 14
  • 21
  • This doesn't address my issue where the resp.headers.get('content-length') = 127 which is not equal to the number of iterations (count=64). The progress bar library and implementation is irrelevant – Jeff Tsui Feb 27 '15 at 19:26
  • 1
    According to requests internals, it automatically decodes gzip. The false content_length while iterating content is hard to debug. – Anton Antonov Feb 27 '15 at 20:06