Questions tagged [dill]

Dill is a module which extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types. Use this tag in conjunction with the pickle and python tag for questions about object serialization with Dill.

Dill extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types.

Major features

Dill can pickle the following standard types:

  • none, type, bool, int, long, float, complex, str, unicode,
  • tuple, list, dict, file, buffer, builtin,
  • both old and new style classes,
  • instances of old and new style classes,
  • set, frozenset, array, functions, exceptions

Dill can also pickle more 'exotic' standard types:

  • functions with yields, nested functions, lambdas
  • cell, method, unboundmethod, module, code, methodwrapper,
  • dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
  • wrapperdescriptor, xrange, slice,
  • notimplemented, ellipsis, quit

Dill cannot yet pickle these standard types:

  • frame, generator, traceback

Dill also provides the capability to:

  • save and load python interpreter sessions
  • save and extract the source code from functions and classes
  • interactively diagnose pickling errors

Current Release

The latest released version of dill is available from http://dev.danse.us/trac/pathos and https://github.com/uqfoundation/dill.

Dill is distributed under a 3-clause BSD license.

290 questions
4
votes
2 answers

pickle error when serialize cython class with lambda

I use pickle and dill for follow lambda function and work fine : import dill import pickle f = lambda x,y: x+y s = pickle.dumps(f) or even when used in class, for example: file foo.py class Foo(object): def __init__(self): self.f =…
Amin
  • 138
  • 1
  • 8
4
votes
1 answer

TypeError: can't pickle PyCapsule objects

I use dill to save ML model to file. When I run my tests with python -m unittest it works. But if I try run tests with python setup.py test it getting error TypeError: can't pickle PyCapsule objects in raw where I try to save model. My settings in…
4
votes
1 answer

How to use dill library for object serialization with shelve library

I'm using PyMemoize library to cache coroutine. I decorated the coroutine, but when Python calls it, I get: TypeError: can't pickle coroutine objects This happens because PyMemoize internally tries to pickle coroutine and store it inside Redis. For…
rominf
  • 2,719
  • 3
  • 21
  • 39
4
votes
1 answer

Conda's version information on package doesn't correspond to __version__

I'm using anaconda (myenv3) foo@foo:~$ which conda /home/foo/anaconda3/bin/conda where in "myenv3" I have dill=2.8.2 installed: (myenv3) foo@foo:~$ conda list -n myenv3 dill # packages in environment at /home/foo/anaconda3/envs/myenv3: # # Name …
FooBar
  • 15,724
  • 19
  • 82
  • 171
4
votes
2 answers

Why does pickling a tensorflow Tensor fail?

Here's a snippet that will succeed in serializing with dill, but fail with pickle. It is surprising that Tensor objects aren't natively pickleable. Is this a fundamental limitation of thread-aware Tensors, or is it just not implemented? import…
eqzx
  • 5,323
  • 4
  • 37
  • 54
4
votes
2 answers

dump and load a dill (pickle) in two different files

I think this is fundamental to many people who know how to deal with pickle. However, I still can't get it very right after trying for a few hours. I have the following code: In the first file import pandas as pd names = ["John", "Mary", "Mary",…
yearntolearn
  • 1,064
  • 2
  • 17
  • 36
4
votes
2 answers

`pickle`: yet another `ImportError: No module named my_module`

I have a class MyClass defined in my_module. MyClass has a method pickle_myself which pickles the instance of the class in question: def pickle_myself(self, pkl_file_path): with open(pkl_file_path, 'w+') as f: pkl.dump(self, f,…
bzm3r
  • 3,113
  • 6
  • 34
  • 67
4
votes
1 answer

Why dill dumps external classes by reference, no matter what?

In the example below, I have placed the class Foo inside its own module foo. Why is the external class dumped by ref? The instance ff is not being dumped with its source code. I am using Python 3.4.3 and dill-0.2.4. import dill import foo class…
obelloc
  • 299
  • 1
  • 14
4
votes
1 answer

Does the dill python module handle importing modules when sys.path differs?

I'm evaluating dill and I want to know if this scenario is handled. I have a case where I successfully import a module in a python process. Can I use dill to serialize and then load that module in a different process that has a different sys.path…
Brent V
  • 43
  • 1
  • 3
4
votes
2 answers

Python Multiprocessing: AttributeError: 'Test' object has no attribute 'get_type'

short short version: I am having trouble parallelizing code which uses instance methods. Longer version: This python code produces the error: Error Traceback (most recent call last): File…
Gil Zellner
  • 889
  • 1
  • 9
  • 20
3
votes
0 answers

How to transform a lambda function into a pickle-able function

One can use the wonderful dill package to pickle lambda functions (amongst others). See this post. Here though, I'd like to dynamically transform lambda functions into object that are functionally equivalent, but are builtin-picklable. (I can give…
thorwhalen
  • 1,920
  • 14
  • 26
3
votes
1 answer

AttributeError: module 'dill._dill' has no attribute 'stack'

Use this colab https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing my CUDA Version: 11.2 when do this train_dataset = train_dataset.map( process_data_to_model_inputs, batched=True, …
JenHao Yang
  • 41
  • 1
  • 3
3
votes
0 answers

Databricks error with dill module. It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation

I am trying to a scikit-learnpipeline on Databricks. However, I encounter the following issue: import dill f = open("/dbfs/HR_pipe.p", mode='wb') dill.dump(fitted,f) where: fitted = main_pipeline.fit(X_train, y_train) Below you can find the error…
3
votes
1 answer

Can't solve "SystemError: unknown opcode"

I am executing a notebook on my laptop and I get the following error. XXX lineno: 17, opcode: 120 --------------------------------------------------------------------------- SystemError Traceback (most recent call…
alec_djinn
  • 10,104
  • 8
  • 46
  • 71
3
votes
2 answers

How can I change the definition of class methods after pickling

I am writing a class that does some data processing. I initiated the class and did some processing then pickled the results. the problem is that I added new methods to the class and loaded the pickled objects but I couldn't apply the new methods to…
Ahmed
  • 43
  • 3
1 2
3
19 20