Create zipfile at local and write files from s3

Question

I am creating a zipfile on my local machine and would like to write files from s3. So far I'm unable to do it. Here's what I have in the mean time.

import os
import zipfile
from fs import open_fs

fs = open_fs(os.getenv('s3_sample_folder'))
file_names = file_names() #list of file names

with zipfile.ZipFile('zipfile.zip', mode='w') as zf:
   for file in file_names:
       with fs.open('/'+file, 'rb') as remote_file:
           content = remote_file.read()
           zf.write(content, basename(content))

blhsing · Accepted Answer · 2019-02-20T17:40:54.433

2

The ZipFile.write method accepts a file name, not file content. You should use the ZipFile.writestr method instead to write file content to the zip file:

zf.writestr(file, content)

edited Feb 20 '19 at 17:40

answered Feb 20 '19 at 17:24

blhsing

91,368
6
71
106

Is it possible to use the `write` method so I can retain the filename? – alvirbismonte Feb 20 '19 at 17:36
The first parameter of `writestr` is filename. Please refer to the [documentation](https://docs.python.org/3/library/zipfile.html#zipfile.ZipFile.writestr) for details. – blhsing Feb 20 '19 at 17:39
You should not use the `write` method because it expects an actual file on the disk, and you only have the file content in the memory. – blhsing Feb 20 '19 at 17:55
Will try this. Thank you. I'm overwhelmed by the docs but will read further coming from your answer. – alvirbismonte Feb 20 '19 at 18:08

score 0 · Answer 2 · answered Feb 25 '19 at 21:30

Since you are using PyFilesystem, you can open a S3 filesystem and a Zip filesystem, then use copy_file to copy between them.

Something like the following should work:

import os
from fs import open_fs
from fs.copy import copy_file

with open_fs(os.getenv('s3_sample_folder')) as s3_fs:
    with open_fs('zip://zipfile.zip', create=True) as zip_fs:
        for filename in file_names():
            copy_file(s3_fs, filename, zip_fs, filename)

Create zipfile at local and write files from s3

2 Answers2