Download files as stream without downloading complete files into memory

Question

I want to download files as stream from a remote server and make them zip stream and return it to frontend as stream without waiting all files to complete downloads.

Is it possible in python and framework of python ?? I tried as below but its not working i am using Django framework:

 zip_buffer = io.BytesIO()

 with zipstream.ZipFile(zip_buffer,"w", zipfile.ZIP_DEFLATED, False) as zf:
     url = "https://download-file/zy"
     r = requests.get(url, stream=True)
     zip_inf = zipstream.ZipInfo(file_name)
     zf.write_iter(zip_inf, r.content)
response = StreamingHttpResponse(zip_buffer.getvalue(), content_type='application/octet-stream')
            response['Content-Disposition'] = 'attachment; filename=zip-files.zip'
            return response

Part of the issue that you didn't loop over the data received from the requests library. — Mohamed ElKalioby, Jun 15 '22 at 08:46
like you want me to read r file stream and send it to the zip ? — Raheela Aslam, Jun 15 '22 at 08:49
Something like the answer https://stackoverflow.com/a/57498146/871410 — Mohamed ElKalioby, Jun 15 '22 at 09:07
i tried and got "ValueError: stat: embedded null character in path" error — Raheela Aslam, Jun 15 '22 at 09:18

Raheela Aslam · Accepted Answer · 2022-06-23T09:42:07.653

I tried by using aiozipstream and fast api for streamingResponse. this solution worked for me.

 files = []
 urls = ['url1','ur2']
 for url in urls:
    files.append({'stream': file_content_generator(url), 'name': url.split('/')[-1]})
 zf = ZipStream(files, chunksize=32768)

 response = StreamingResponse(zf.stream(), media_type='application/zip')
    return response

def file_content_generator(url):
     with httpx.stream('GET', url, timeout=None) as r:
          yield from r.iter_bytes()

Download files as stream without downloading complete files into memory

1 Answers1