7

I have a bunch of objects created from classes imported with

module = imp.load_source(packageName, packagePath)

that I need to pickle. It all works perfectly, as long as packagePath is directly in the Python path or the working dir.

But as soon as I move it somewhere else, I get the dreaded

ImportError: No module named test_package

I tried adding a __reduce__ method that returns the class as first value. I tried using dill, which is supposedly able to serialize full classes, and not a simple reference to the class (and I tried combining it with __reduce__).

The way it works currently, is by double-pickling them along with the package path in an object that takes care of importing the package:

class Container(object):

    def __init__(self, packagePath, packageName, objectsDump= None):
        self.package = imp.load_source(packageName, packagePath)
        self.packagePath = packagePath
        self.packageName= packageName
        if objectsDump is not None:
            self.objects = dill.loads(objectsDump)

    def __reduce__(self):
        return (self.__class__,
               (self.packagePath, self.packageName, dill.dumps(self.objects))

I find this way really convoluted, and I would like to know: Is there a more pythonic way to acheive this?

Note: all of this happens in Python 2.7.10, dill 0.2.6. All objects to serialize are new-style objects (inherit from object).

  • For `dill`, this exact problem is currently being discussed in this [GitHub issue](https://github.com/uqfoundation/dill/issues/123#issuecomment-1214696766). Both `dill` and `pickle` expect that, if a module is available in the "pickling" interpreter, it also should be available in the "unpickling" interpreter. Currently, only if the class' module can't be located (e.g. if its `__module__` attribute is set to `None`) is that `dill` saves the class by value instead of by reference. Changes in this behavior are also being discussed [here](https://github.com/uqfoundation/dill/issues/424). – leogama Aug 17 '22 at 00:39
  • * Classes defined in the `__main__` module are also saved by value in `dill`. – leogama Aug 17 '22 at 01:06

0 Answers0