2

I have a script which creates a closed in-memory ZipFile object that I need to post as a bytestring (using requests); how do I do that? I have tried opening the file, which fails with "TypeError: expected str, bytes or os.PathLike object, not ZipFile"

The script works just fine if I write the ZipFile to a file and then open that file for the post data. However it will probably iterate over a couple million files, and that seems like a lot of temp files, and disk activity.

import io
import zipfile
from PIL import Image

z = io.BytesIO()
zfile = zipfile.ZipFile(z,"a")

zipdict = {}

img_loc = "D:/Images/seasons-3.jpg"
im_original = Image.open(img_loc)
imfmt = im_original.format
im = im_original.copy()
im_original.close()
im_out = io.BytesIO()
im.save(im_out,imfmt)
zfile.writestr("seasons-3.jpg",im_out.getvalue())
im_out.close()
zipdict['seasons-3']=zfile
zfile.close()

running with error:

Python 3.6.3 (v3.6.3:2c5fed8, Oct  3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> 
>>> zipdict['seasons-3']
<zipfile.ZipFile [closed]>
>>> pl_data = open(zipdict['seasons-3'])
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    pl_data = open(zipdict['seasons-3'])
TypeError: expected str, bytes or os.PathLike object, not ZipFile
>>>
Dharman
  • 30,962
  • 25
  • 85
  • 135
Tim Achee
  • 37
  • 1
  • 3
  • 1
    Can you paste your code and error traceback? – Cheche Nov 20 '18 at 16:51
  • 2
    "closed in-memory ZipFile" - you're going to have to explain that some more. Did you wrap a ZipFile around a BytesIO or something? – user2357112 Nov 20 '18 at 16:51
  • I'm not sure if it helps with an in-memory zip-file (never encountered one in the wild), but you can unzip a single file from an archive: https://stackoverflow.com/a/46423414/962190 – Arne Nov 20 '18 at 16:56
  • @user2357112 that's pretty much exactly what I did. I created a ZipFile and used writestr to add a couple of BytesIO to the ZipFile. Then I added the ZipFile as value to a dict, with key as filename, and closed the ZipFile. – Tim Achee Nov 21 '18 at 07:13
  • @TimAchee: Nope, that still doesn't explain things. What, if anything, did you do to put the ZipFile itself in memory? What arguments did you pass to the ZipFile constructor? – user2357112 Nov 21 '18 at 07:33
  • @user2357112 I added the ZipFile to a dict to keep it in-memory, and passed the filename and data as arguments. – Tim Achee Nov 21 '18 at 08:06
  • @Cheche sorry about that, took a bit to remove the irrelevant code. I just edited the post with the code and error. – Tim Achee Nov 21 '18 at 08:23
  • @Arne Thank you Arne, but I don't have a problem getting the contents out of the zip, I need to turn the entire zip into a bytestream – Tim Achee Nov 21 '18 at 08:26
  • "I added the ZipFile to a dict to keep it in-memory" - that's not how in-memory files work. That step isn't necessary. – user2357112 Nov 21 '18 at 08:51

1 Answers1

0

zfile is closed. It's useless to you. The thing you need to use now is z, the file-like object that was managing the underlying binary storage for the ZipFile.

You can use z.getvalue() to get a bytestring representing the contents of z, just like you did with im_out, or you can seek back to the beginning with z.seek(0) and use it with the parts of requests that take file-like objects.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • thank you user2357112, but I need to post the data as a bytestring of a zipped file so I'm not sure this solution would work out. – Tim Achee Nov 21 '18 at 09:20
  • @TimAchee: What makes you think this wouldn't work? It sounds like you're misunderstanding the role of the ZipFile object. – user2357112 Nov 21 '18 at 19:21
  • thank you again and sorry for the delay in responding. I second guessed myself yesterday morning and came to pretty much the same conclusion. So I tried uploading z.getvalue() as my data. While the file uploaded to the server, the system I was trying to ingest into rejected it with "Error (archive is not a ZIP archive) occurred while unzipping file" – Tim Achee Nov 22 '18 at 07:35
  • @TimAchee: Could be a problem with how you're interacting with the web API, or perhaps you accidentally put something else into `z` besides the zip contents. It's hard to tell from here. – user2357112 Nov 22 '18 at 07:52
  • yeah, when I run the script saving the zip to disk and using open(path_to_saved_zip) instead of z.getvalue(), everything works. If I understand you correctly, z.getvalue() should be equivalent to open(saved_zip), right? – Tim Achee Nov 22 '18 at 08:46
  • @TimAchee: It should be more like `open(saved_zip, 'rb').read()`, where `open` is the built-in `open`. – user2357112 Nov 22 '18 at 18:35
  • ah, and that's the rub. I think I need to find the equivalent to open(saved_zip) without having an actual saved zip. – Tim Achee Nov 23 '18 at 07:10
  • @TimAchee: `z.seek(0)` and use `z`. – user2357112 Nov 23 '18 at 07:11
  • thank you very much for sticking with me this far. Even after using the seek(0) method on z, uploading z as my data still causes the ingest process to error with "Error (archive is not a ZIP archive) occurred while unzipping file ". – Tim Achee Nov 23 '18 at 09:41