How to dump a function with import inside

Question

I am trying to dump functions into a file, so that I can use this file/function elsewhere. I choose dill rather than pickle, because I need dependencies. However, dill doesn't work if the function has imports inside. For example:

def func():
    import numpy

import dill
dill.settings['recurse'] = True 
with open("test.pickle","wb") as f:
    dill.dump(func,f)

When I restart and load the function back, I get this error,

import dill 
func = dill.load(open("test.pickle"))
func()
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input> in <module>()
      1 import dill
      2 func = dill.load(open("test.pickle"))
----> 3 func()

<ipython-input> in func()

ImportError: __import__ not found

This example works if I use pickle to dump, but pickle seems not to save dependencies recursively, so I can't save functions like def fun1(): return fun2(). Is there a way to dump functions with both import and dependencies? I feel pickle or dill only does half.

score 2 · Accepted Answer · answered Apr 27 '20 at 12:43

I'm the dill author. I believe dill should also work for you:

$ python
Python 3.6.10 (default, Dec 21 2019, 11:39:07) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> def func():
...   import numpy
... 
>>> import dill
>>> 
>>> with open('XXX.pkl', 'wb') as f:
...   dill.dump(func, f)
... 
>>> 

$ python
Python 3.6.10 (default, Dec 21 2019, 11:39:07) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> func = dill.load(open('XXX.pkl', 'rb'))
>>> func()
>>>

The recurse settings means recursively trace references through the global dict, but don't store the entire global dict. The default setting for dill is to store all of the global dict when pickling a function. So, recurse can make the pickles smaller, but they can also fail due to missing references.

How to dump a function with import inside

1 Answers1