I'm trying to stream a download from an nginx server and simultaneously upload it. The download is using requests stream implementation; the upload is using chunking - the intention is to be able to report progress as the down/upload is occurring.
The overall code of what I've got so far is like so:
with closing(requests.get(vmdk_url, stream=True, timeout=60 + 1)) as vmdk_request:
chunk_in_bytes = 50 * 1024 * 1024
total_length = int(vmdk_request.headers['Content-Length'])
def vmdk_streamer():
sent_length = 0
for data in vmdk_request.iter_content(chunk_in_bytes):
sent_length += len(data)
progress_in_percent = (sent_length / (total_length * 1.0)) * 100
lease.HttpNfcLeaseProgress(int(progress_in_percent))
yield data
result = requests.post(
upload_url, data=vmdk_streamer(), verify=False,
headers={'Content-Type': 'application/x-vnd.vmware-streamVmdk'})
Which, in a certain set of contexts, works fine. I put it into another (a Cloudify plugin, if you're interested) and when it reaches around 60s it fails to read data.
So I'm looking for an alternative - or simply better - way of streaming a download/upload as my 60s issue might revolve around how I'm streaming (I hope). Preferably with requests but really I'd use anything up to and including raw urllib3.
