Stream Files to Zip File in Azure Blob Storage using Python?

Question

I have the following problem in Python:

I am looking to create a zipfile in Blob Storage consisting of files from an array of URLs but I don't want to create the entire zipfile in memory and then upload it. I ideally want to stream the files to the zipfile in blob storage. I found this write up for C# https://andrewstevens.dev/posts/stream-files-to-zip-file-in-azure-blob-storage/ as well as this answer also in C# https://stackoverflow.com/a/54767264/10550055 .

I haven't been able to find equivalent functionality in the python azure blob SDK and python zipfile library.

Did you manage to find the solution to this? – Birat Bade Shrestha Feb 24 '22 at 10:50 — Birat Bade Shrestha, Feb 24 '22 at 10:50

score 0 · Answer 1 · answered May 22 '21 at 07:15

Try this :

from zipfile import ZipFile
from azure.storage.blob import BlobServiceClient
import os,requests


tempPath = '<temp path>'

if not os.path.isdir(tempPath):
    os.mkdir(tempPath)

zipFileName = 'test.zip'

storageConnstr = ''
container = ''

blob = BlobServiceClient.from_connection_string(storageConnstr).get_container_client(container).get_blob_client(zipFileName)


fileURLs = {'https://cdn.pixabay.com/photo/2015/04/23/22/00/tree-736885__480.jpg',
'http://1812.img.pp.sohu.com.cn/images/blog/2009/11/18/18/8/125b6560a6ag214.jpg',
'http://513.img.pp.sohu.com.cn/images/blog/2009/11/18/18/27/125b6541abcg215.jpg'}



def download_url(url, save_path, chunk_size=128):
    r = requests.get(url, stream=True)
    with open(save_path, 'wb') as fd:
        for chunk in r.iter_content(chunk_size=chunk_size):
            fd.write(chunk)

zipObj = ZipFile(tempPath + zipFileName, 'w')

#download file and write to zip
for url in fileURLs:
    localFilePath = tempPath + os.path.basename(url)
    download_url(url,localFilePath)
    zipObj.write(localFilePath)
    
zipObj.close()

#upload zip
with open(tempPath + zipFileName, 'rb') as stream:
    blob.upload_blob(stream)

Thanks for your help! I should probably have been more clear in my question. What I would like to achieve is to never construct the zipfile in local storage but to stream the files directly to a new zipfile located in blob storage. That is what is accomplished in the C# examples as far as I can tell. I plan on running this script as an Azure function. — Rawzm, May 22 '21 at 12:52
@Rawzm, welcome, actually you can save temp files on Azure Function, it has own file system, details see this post :https://stackoverflow.com/questions/63265669/is-possible-to-save-a-temporaly-file-in-a-azure-function-linux-consuption-plan-i . What's more, if a zip file size larger than 1.5 GB, it can't be loaded into Azure function memory as function memory limit is 1.5GB — Stanley Gong, May 22 '21 at 13:36

Stream Files to Zip File in Azure Blob Storage using Python?

1 Answers1