Questions tagged [dill]

Dill is a module which extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types. Use this tag in conjunction with the pickle and python tag for questions about object serialization with Dill.

Dill extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types.

Major features

Dill can pickle the following standard types:

  • none, type, bool, int, long, float, complex, str, unicode,
  • tuple, list, dict, file, buffer, builtin,
  • both old and new style classes,
  • instances of old and new style classes,
  • set, frozenset, array, functions, exceptions

Dill can also pickle more 'exotic' standard types:

  • functions with yields, nested functions, lambdas
  • cell, method, unboundmethod, module, code, methodwrapper,
  • dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
  • wrapperdescriptor, xrange, slice,
  • notimplemented, ellipsis, quit

Dill cannot yet pickle these standard types:

  • frame, generator, traceback

Dill also provides the capability to:

  • save and load python interpreter sessions
  • save and extract the source code from functions and classes
  • interactively diagnose pickling errors

Current Release

The latest released version of dill is available from http://dev.danse.us/trac/pathos and https://github.com/uqfoundation/dill.

Dill is distributed under a 3-clause BSD license.

290 questions
0
votes
2 answers

How to speed up dill serialization to store Python object to file

It says in the documentation that the output of sys.getsizeof() is in bytes. I'm trying to store a data structure that is a dictionary of class instances and lists. I did sys.getsizeof() on this dictionary of class instances and it was 3352 bytes. …
O.rka
  • 29,847
  • 68
  • 194
  • 309
0
votes
0 answers

If Dill file is too large for RAM is there another way it can be loaded

If a dill file is to large to RAM, is it possible to load it in an alternative method. For example python3 is throwing Memory Errors when I load a serialized object of about 1.2 GB. file = open('SerializedData.pkl', 'rb') data =…
Troll_Hunter
  • 465
  • 2
  • 9
  • 15
0
votes
1 answer

Pickle Error when loading an object in python?

When I try to load a file in python I get this error from dill: Traceback (most recent call last): File "LArSoftSGD.py", line 154, in stat_bundle = train_batch_iterator(clf, TOTAL_TRAINED_EVENTS) File "LArSoftSGD.py", line 121, in…
Troll_Hunter
  • 465
  • 2
  • 9
  • 15
0
votes
1 answer

returning a two dimensional array by multiprocessing

In the following code which is an example of my main code, I have tried to use pathos.multiprocessing to increase the speed of iteration of a loop. The output of each iteration which has implemented with multiprocessing is a 2-D array. I used…
Dalek
  • 4,168
  • 11
  • 48
  • 100
0
votes
1 answer

Python: Trouble with dill installation

I am currently working on a big game project in python and the need for storing my classes has just appeared. I looked into the net and tried out pickle and cpickle but given the structure of my classes they weren't good enough. so I am currently…
PHD
  • 3
  • 3
0
votes
1 answer

Save the optimization results from PyGMO by using pickle or dill

I generated a population using PyGMO. A population is a class which contains the individual results of the computation. I can iterate through the population and save the current function values and the parameter values. Unfortunately I cannot dump…
Moritz
  • 5,130
  • 10
  • 40
  • 81
0
votes
2 answers

WindowsError in pathos.multiprocessing

import subprocess import os import dill import pathos.multiprocessing as mp def main(processes): list_of_files = os.listdir(os.getcwd()) pool = mp.ProcessingPool(processes) pool.map(subprocess.call, [['program.exe',fname] for…
wye_naught
  • 33
  • 4
0
votes
0 answers

Using dill to pickle workflow objects

I'm trying to pickle (using the dill extension) a workflow object from pyutilib.workflow like so using python 2.7. The end objective here is to be able to insert these workflow objects into a MongoDB database and pulled out on the other end when…
Connor G.
  • 23
  • 1
  • 4
0
votes
1 answer

Python multiprocess change class instance in place

I have a list of class instances, and I want to call the same instance method in parallel, use pathos to be able to pickle instance method, The true problem is when I want to change/add an attribute to the instances, it doesn't work, I think this is…
fast tooth
  • 2,317
  • 4
  • 25
  • 34
0
votes
1 answer

Good example of BadItem in Dill Module

I'm exploring the detect method of Dill and am looking for a good simple example of a bad item - one that's unpicklable by Dill. I first thought of a process and tried: >>> proc = os.popen('ls -l') >>> proc
MikeiLL
  • 6,282
  • 5
  • 37
  • 68
0
votes
1 answer

Getting "ImportError: No module named" with parallel python and methods in a package

I'm trying to use parallel python in order to do some distributed benchmarking (essentially, coordinate and run some code on a set of machines from a central server). The code I had was working perfectly fine until I moved the functionality to a…
Nikola Knezevic
  • 789
  • 5
  • 20
0
votes
1 answer

Tell IPython Parallel to use Pickle again after Dill has been activated

I'm developing a distributed application using IPython parallel. There are several tasks which are carried out one after another on the IPython cluster engines. One of these tasks inevitably makes use of closures. Hence, I have to tell IPython to…
t3c
  • 3
  • 4
0
votes
2 answers

dill.detect.at Cannot reference object at '0x1023d2600'

More digging into dill. Specifically the detect.at method which is a call to: def _locate_object(address, module=None): """get object located at the given memory address (inverse of id(obj))""" special = [None, True, False] #XXX: more...? …
MikeiLL
  • 6,282
  • 5
  • 37
  • 68
0
votes
1 answer

Dill.detect.children object types

Dill.detect.children requires two arguments; obj and objtype. Inspecting an audiofile object I can call: dill.detect.children(audiofile, object) dill.detect.children(audiofile, dict) dill.detect.children(audiofile, list) Which return without…
MikeiLL
  • 6,282
  • 5
  • 37
  • 68
0
votes
1 answer

pickle error assert id(obj) not in self.memo

I am using dill (advanced version of pickle) right now. I want to serialize my object, but I get this error: /usr/lib/python2.7/pickle.pyc in memoize(self, obj) 242 if self.fast: 243 return --> 244 assert id(obj)…
psuedobot
  • 141
  • 3
  • 11
1 2 3
19
20