0

I have a directory containing mostly text and json files, and one binary file (output of MXNet.Block.save_parameters neural network).

I wanted to zip this folder and then pickle it. Say I have a zip file object:

from zipfile import ZipFile
import os, pickle, itertools


files = list(itertools.chain(*[
    map(lambda x: os.path.join(root, x), files)
    for root, directories, files in os.walk('model-artifacts/')
]))

zfile = ZipFile('mymode.l.zip', 'w')

for file in file_paths:
   zfile.write(file)

I cannot really pickle it:

pickle.dumps(zfile)

# TypeError: cannot serialize '_io.BufferedRandom' object

I was wondering if there is a way to pickle zipfiles or any way to pickle contents of a directory.

WHY?

I am not doing the pickling myself, but using a library Metaflow which pickles objects in it, so I want to find a way to store my model with Metaflow

Gerges
  • 6,269
  • 2
  • 22
  • 44
  • You're writing your zipped data to a path on disk. What do you want the pickle to represent? Should it represent a file path? Should it represent the zipped data? If it's the second option, why are you writing to disk? – user2357112 Dec 08 '20 at 01:34
  • Yes you are right :( I don't think I know what I am doing here, I may need to try to understand pickle a little better... The problem is I cannot pickle my `model` object directly, its giving an error: `AttributeError: Can't pickle local object 'SimpleFeedForwardNetworkBase.__init__..' ` . I thought if I load only the model artifacts instead (json files and a binary files), then the framework I am using will know how to pickle it, and then I know how to handle that downstream... – Gerges Dec 08 '20 at 01:41
  • as another side note: The framework I am using (`Metaflow`) pickles objects and uploads them to the cloud. I can do it myself, but I prefer to have everything accessible with the framework. – Gerges Dec 08 '20 at 01:44
  • Did you write your model class yourself? If so, look at the `__init__` method and convert lambdas to module-level helper functions. – user2357112 Dec 08 '20 at 01:44
  • No :(. This is a class from `MXNet`, can't really change it. – Gerges Dec 08 '20 at 01:45
  • I don't see a `SimpleFeedForwardNetworkBase` in MXNet. – user2357112 Dec 08 '20 at 01:47
  • Sorry .. I should be more precise. It is `gluon-ts`, [here's the code](https://github.com/awslabs/gluon-ts/blob/master/src/gluonts/model/simple_feedforward/_network.py#L25). It uses MXNet under the hood to save the model. – Gerges Dec 08 '20 at 01:49

1 Answers1

1

Short answer: You cannot pickle a Zip object.

Explanation: A Zip file is a file which has been compressed. The purpose of pickling is that we are trying to serialize some (python) object. However, after file compression, you don't really have (python) objects anymore, simply a bunch of 0's and 1's ready to be decompressed.

tcglezen
  • 456
  • 1
  • 4
  • 12