Stack Overflow
Stack Overflow

Questions tagged [dill]

Dill is a module which extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types. Use this tag in conjunction with the pickle and python tag for questions about object serialization with Dill.

Dill extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types.

Major features

Dill can pickle the following standard types:

  • none, type, bool, int, long, float, complex, str, unicode,
  • tuple, list, dict, file, buffer, builtin,
  • both old and new style classes,
  • instances of old and new style classes,
  • set, frozenset, array, functions, exceptions

Dill can also pickle more 'exotic' standard types:

  • functions with yields, nested functions, lambdas
  • cell, method, unboundmethod, module, code, methodwrapper,
  • dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
  • wrapperdescriptor, xrange, slice,
  • notimplemented, ellipsis, quit

Dill cannot yet pickle these standard types:

  • frame, generator, traceback

Dill also provides the capability to:

  • save and load python interpreter sessions
  • save and extract the source code from functions and classes
  • interactively diagnose pickling errors

Current Release

The latest released version of dill is available from http://dev.danse.us/trac/pathos and https://github.com/uqfoundation/dill.

Dill is distributed under a 3-clause BSD license.

290 questions
2
votes
1 answer

Python3: pickle a function without side effects

I have a project with a function foo in a module my_project.my_functions. I want to pickle that function in a way that I can unpickle it from somewhere else without requiring to import my_project. foo does not have any side effect, so no…
user3091275
  • 1,013
  • 2
  • 11
  • 27
2
votes
1 answer

Python multiprocess dict of list

I need to do some stuffs in multiprocess with Python 3.6. Namely, I have to update a dict adding lists of objects. Since these objects are unpickable I need to use dill instead of pickle and multiprocess from pathos instead of multiprocessing, but…
fortea
  • 345
  • 4
  • 15
2
votes
0 answers

Using dill to pickle a class instance and loading in a different package

I am trying to use dill to share objects between two different environments (machine learning research and production). A simplified example: package_a/module_a.py: class P: def __init__(self, a, b): self.a = a #for example a dict self.b =…
Valentin Zambelli
  • 149
  • 1
  • 10
2
votes
0 answers

Pathos multiprocessing pool.map not honoring recurse=True

When I try to use pool.map on a lambdified function from sympy.lambdify, dill throws me an error: In [1]: import sympy as sy In [2]: from sympy.abc import x In [3]: f = sy.lambdify(x,x+x,'numpy') In [4]: from pathos.multiprocessing import…
Andrea Olivo
  • 21
  • 5
2
votes
1 answer

Python - dill: Can't pickle decorated class

I have the following code, which decorates the class: import dill from collections import namedtuple from multiprocessing import Process def proxified(to_be_proxied): b = namedtuple('d', []) class Proxy(to_be_proxied, b): pass …
carbolymer
  • 1,439
  • 1
  • 15
  • 30
2
votes
1 answer

Why can't python module dill pickle the generator function?

Dill module is awesome, it can pickle the entire python interpretor global state. But Dill cannot pickle generators. I know it is not supported. But i would like to know the technical reason given that it can pickle python functions?
AravindR
  • 677
  • 4
  • 11
2
votes
2 answers

Python changing ClassA instance variables in ClassB

I am trying to load a whole class instance via dill rather than dump and load each class variable one at a time. Can anybody show me how to do this: class Object(object): pass class ClassA: def __init__(self): self.DATA =…
someuser
  • 2,279
  • 4
  • 28
  • 39
2
votes
1 answer

Why can't dill/pickle class definition?

dill is a great tool for pickling most the Python objects, I use it in IPython parallel to serialize calculations. One issue I keep getting into is around dill-ing class definitions. One of the errors I get is explained below. While trying to…
Mark Horvath
  • 1,136
  • 1
  • 9
  • 24
2
votes
0 answers

Why do I get a module has no attribute __main__ in ipython parallel gather statement?

This is a related question to this one: https://stackoverflow.com/questions/27596463/why-do-i-get-a-cannedarray-object-has-no-attribute-pickled-in-ipython-gather The difference is that now I'm trying to use_dill in ipython parallel and when I try to…
evan54
  • 3,585
  • 5
  • 34
  • 61
2
votes
1 answer

Example of a chain of referents, referrers

Working further with the Dill package. Posted yesterday exemplifying some ignorance of referents, referrers, inheritance and containers. The answer was insightful, but I'm still having trouble coming up with examples that show a few levels of depth…
MikeiLL
  • 6,282
  • 5
  • 37
  • 68
2
votes
0 answers

Multiprocessing Large Objects Using Pathos in Python

I am trying to make use of my computer's multiple CPUs. However, the BeautifulSoup object returned by my function as part of an SQLAlchemy object is not picklable with pickle or cPickle so I am using pathos, a fork of the multiprocssing package…
Michael
  • 13,244
  • 23
  • 67
  • 115
2
votes
1 answer

dill dump_session with ipython

I'm trying to use the dill module to save my ipython session using dump_session() but I'm getting an error message. I'm using Ipython 1.0.0 and dill 0.2-a-dev 20120503. Does anyone out there have any insight? Thanks in advance. Niall Here's the…
nrob
  • 861
  • 1
  • 8
  • 22
1
vote
0 answers

How do use dill to store inputs to a python module also used by others as a submodule?

I develop a python module, MyModule. The user adds it as a git submodule, and imports multiple classes and functions. from MyModule.MyModuleSubfolder import ClassA from MyModule.MyModuleSubfolder import ClassB from MyModule.MyModuleOtherSubfolder…
Shim'on
  • 11
  • 2
1
vote
1 answer

pickle and dill can't load objects with overridden __hash__ function (AttributeError)

In the next few lines of code I'll replicate on a smaller scale what's happening with my program. Class A must store a dictionary with keys that have type A (values can be any type to replicate the error). class A: def __init__(self, name): …
simone
  • 13
  • 2
1
vote
1 answer

AttributeError: module 'dill' has no attribute 'extend'

I have just installed Pytorch, using: (base) C:\>pip3 install torch torchvision torchaudio Requirement already satisfied: torch in c:\users\Emil\appdata\roaming\python\python38\site-packages (1.9.0) Requirement already satisfied: torchvision in…
Emil
  • 1,531
  • 3
  • 22
  • 47
1 2 3
…
19 20