Stack Overflow
Stack Overflow

Questions tagged [dill]

Dill is a module which extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types. Use this tag in conjunction with the pickle and python tag for questions about object serialization with Dill.

Dill extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types.

Major features

Dill can pickle the following standard types:

  • none, type, bool, int, long, float, complex, str, unicode,
  • tuple, list, dict, file, buffer, builtin,
  • both old and new style classes,
  • instances of old and new style classes,
  • set, frozenset, array, functions, exceptions

Dill can also pickle more 'exotic' standard types:

  • functions with yields, nested functions, lambdas
  • cell, method, unboundmethod, module, code, methodwrapper,
  • dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
  • wrapperdescriptor, xrange, slice,
  • notimplemented, ellipsis, quit

Dill cannot yet pickle these standard types:

  • frame, generator, traceback

Dill also provides the capability to:

  • save and load python interpreter sessions
  • save and extract the source code from functions and classes
  • interactively diagnose pickling errors

Current Release

The latest released version of dill is available from http://dev.danse.us/trac/pathos and https://github.com/uqfoundation/dill.

Dill is distributed under a 3-clause BSD license.

290 questions
3
votes
2 answers

Reading file from Azure blob storage using pickle or dill without saving to disk

I'm trying to read weights for a machine learning model from Azure Storage Blob in Python. This should be running in Azure Functions, so I don't believe I'm able to use methods which save the blob to disk. I'm using azure-storage-blob 12.5.0, not…
Samuel
  • 211
  • 2
  • 10
3
votes
2 answers

dill.dump_session won't work in jupyter lab

I want to start working on jupyter lab instead of spyder but i have a problem. I can't save the variables in my workspace. I am trying to use jupyter lab to run a code like this: import dill import pandas as pd import numpy as np df =…
Dan Aizenberg
  • 69
  • 7
3
votes
3 answers

Getting a hash of a function that is stable across runs

IIUC python hash of functions (e.g. for use as keys in dict) is not stable across runs. Can something like dill or other libraries be used to get a hash of a function which is stable across runs and different computers? (id is of course not stable).
Uri
  • 25,622
  • 10
  • 45
  • 72
3
votes
1 answer

Pickling Cython decorated function results in PicklingError

I have the following code: def decorator(func): @functools.wraps(func) def other_func(): print('other func') return other_func @decorator def func(): pass If I try to pickle func everything works. However if I compile the…
stefano1
  • 161
  • 10
3
votes
1 answer

Can dill remember libraries used by a class?

If I create a class that imports a library and use dill to pickle it, when I unpickle it I cannot find the library: import dill from sklearn.metrics.cluster import adjusted_rand_score import pandas as pd import random class Test1(): def…
ojunk
  • 879
  • 8
  • 21
3
votes
1 answer

dill/pickle issue with built-in function

When using dill to serialize a class: import dill, pickle class project(object): def __init__(self, name='', folder='', user_id='', version=-1 ): self.name, self.folder, self.user_id, self.version = name, folder, user_id, version #…
tensor
  • 3,088
  • 8
  • 37
  • 71
3
votes
2 answers

mpi4py: Replace built-in serialization

I'd like to replace MPI4PY's built-in Pickle-serialization with dill. According to the doc the class _p_Pickle should have 2 attributes called dumps and loads. However, python says there are no such attributes when i try the following from mpi4py…
OD IUM
  • 1,555
  • 2
  • 16
  • 26
2
votes
1 answer

Not able to use map() or select(range()) with Huggingface Dataset library, gives dill_.dill has no attribute log

I'm not able to do dataset.map() or dataset.select(range(10)) with huggingface Datasets library in colab. It says dill_.dill has no attribute log I have tried with different dill versions, but no luck. I tried with older versions of dill lib but…
user21401461
  • 41
  • 3
2
votes
1 answer

custom imports from tuning step script - sagemaker pipelines

I have a tuning step in my sagemaker pipeline, in the following step i'm using train.py script inside the tuning step container. Inside the train.py script i'm using imported module called 'dill'. It seems that the sagemaker SKLearn container didn't…
Ron Fisher
  • 43
  • 5
2
votes
1 answer

AttributeError: module 'dill._dill' has no attribute 'log'

I am using a python nlp module to train a dataset and ran into the following error: File "/usr/local/lib/python3.9/site-packages/nlp/utils/py_utils.py", line 297, in save_code dill._dill.log.info("Co: %s" % obj) AttributeError: module…
Fernando
  • 57
  • 1
  • 4
2
votes
1 answer

Unable to load pickled custom estimator sklearn pipeline

I have a sklearn pipeline that uses custom column transformer, estimator and different lambda functions. Because Pickle cannot serialize the lambda functions, I am using dill. Here is the custom estimator I have: class customOLS(BaseEstimator): …
Obiii
  • 698
  • 1
  • 6
  • 26
2
votes
1 answer

Dill cannot deserialize in another environment

Overview I need an Azure python function to serve up a pickled object, and I need the resultant pkl file to be reasonably portable, so that it can be used in any typical notebook or python code file with minimal imports. Test code used to generate…
RTD
  • 334
  • 2
  • 12
2
votes
1 answer

Integrating Flask/Dill to dump/load server sessions

I'm trying to integrate Flask with Dill to dump/load Python sessions on the server side. The code below has two functions, the first one sets the value of x to zero and imports the datetime library. The second one increments x by 1 and gets the…
ps0604
  • 1,227
  • 23
  • 133
  • 330
2
votes
1 answer

Conflict between dill and pickle while using pyspark

Currently pyspark uses 2.4.0 version as part of conda installation. pip installation allows to use a later version of pyspark which is 3.1.2. but using this version, dill library has conflicts with pickle library. i use this for unit test for…
python_enthusiast
  • 21
  • 3
2
votes
0 answers

Dill de-serialize NMF child class fails

Summary I created a child class extending sklearn's NMF. Pickling it and unpickling an instance with dill fails if I don't re-define the class before. Minimal (not) Working Example First, create some custom classes and instances, and pickle them: #…
Battleman
  • 392
  • 2
  • 12
1 2 3
…
19 20