Questions tagged [dill]

Dill is a module which extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types. Use this tag in conjunction with the pickle and python tag for questions about object serialization with Dill.

Dill extends python's 'pickle' module for serializing and de-serializing python objects to the majority of the built-in python types.

Major features

Dill can pickle the following standard types:

  • none, type, bool, int, long, float, complex, str, unicode,
  • tuple, list, dict, file, buffer, builtin,
  • both old and new style classes,
  • instances of old and new style classes,
  • set, frozenset, array, functions, exceptions

Dill can also pickle more 'exotic' standard types:

  • functions with yields, nested functions, lambdas
  • cell, method, unboundmethod, module, code, methodwrapper,
  • dictproxy, methoddescriptor, getsetdescriptor, memberdescriptor,
  • wrapperdescriptor, xrange, slice,
  • notimplemented, ellipsis, quit

Dill cannot yet pickle these standard types:

  • frame, generator, traceback

Dill also provides the capability to:

  • save and load python interpreter sessions
  • save and extract the source code from functions and classes
  • interactively diagnose pickling errors

Current Release

The latest released version of dill is available from http://dev.danse.us/trac/pathos and https://github.com/uqfoundation/dill.

Dill is distributed under a 3-clause BSD license.

290 questions
1
vote
1 answer

Python: Pickle class object that has functions/callables as attributes

I have custom class which inherits from functools.partial from functools import partial from typing import Callable class CustomPartial(partial): def __new__(cls, func_name: str, func: Callable, *args, **kwargs): …
1
vote
2 answers

Is there a way to not use Pickling when using the python multi processing module?

I am having a very hard time figuring this out. Im trying to create a real time satellite tracker and using the sky field python module. It reads in TLE data and then gives a LAT and LON position relative to the earth. The sky filed module creates…
Koss24
  • 11
  • 1
1
vote
0 answers

Client Cancelled Exception when running Apache FlinkRunner via HuggingFace Datasets

I am trying to download wikipedia corpora through HuggingFace's Datasets library. Most of the languages I've needed have downloaded successfully, but I cannot seem to get Cebuano, Spanish, or Russian to finish downloading. Here is an example of how…
1
vote
1 answer

Can pickle/dill `foo` but not `lambda x: foo(x)`

I am applying some preprocessing to the CIFAR100 dataset from datasets.load import load_dataset from datasets import Features, Array3D from transformers.models.vit.feature_extraction_vit import ViTFeatureExtractor # Resampling &…
nalzok
  • 14,965
  • 21
  • 72
  • 139
1
vote
2 answers

Dill keeps previous loaded session when loading new session

I'm generating 2 session dumps with import dill a = 1234 dill.dump_session('test1.session') and import dill b = 1234 dill.dump_session('test2.session') Then loading them back into a module sequentially import types import dill mod =…
wasp256
  • 5,943
  • 12
  • 72
  • 119
1
vote
0 answers

Dill sometimes makes attributes of objects disappear

I have a class 1 (Protpkg) with a custom-defined __new__ method that uses an object of a different custom class 2 (State) as argument and saves it as instance attribute. Then, the initialization (__init__) of this class 1 uses a multiprocess Pool to…
1
vote
0 answers

Math behind dill and pickle

What is the math behind serialization and deserialization behind the dill and pickle libraries? Why dill can serialize/deserialization more objects than pickle? I could not find the serialize/deserialization in their github…
Ngo Cuong
  • 63
  • 5
1
vote
1 answer

Dill doesn't seem to respect metaclass

I have recently begun working with dill. I have a metaclass, which I use to create a Singleton pattern. Always have one object at any instant. I am using dill to serialise.The problem is once the object is loaded back, it doesn't respect the…
Aaron
  • 13
  • 3
1
vote
0 answers

Multiprocessing with PySwig Object

PySwig is a C++ wrapper which creates objects which (for unavoidable reasons) cannot be pickled by Python. I want to access a method on that object to run across a large dataset (around 1M entries) in parallel. I can run a function with multiple…
Tarquinnn
  • 501
  • 3
  • 8
1
vote
0 answers

How to open a pickled file with dill where the objects are not findable anymore?

I saved a bunch of stuff in one single pickle file with dill and some of the data are objects. Unfortunately, I didn't realize that if I changed my code structure e.g., heavy refactoring that dill wouldn't be able to find my code anymore for loading…
Charlie Parker
  • 5,884
  • 57
  • 198
  • 323
1
vote
0 answers

Dill TypeError: cannot pickle 'sqlite3.Connection' object. No Sqlite anywhere in function

I am getting TypeError: cannot pickle 'sqlite3.Connection' object when attempting to pickle a function. The thing is, I don't use sqlite anywhere in my project and definitely nowhere in this function. Here is a minimal working example that shows…
skrhee
  • 336
  • 5
  • 19
1
vote
1 answer

ERROR:root:can't pickle fasttext_pybind.fasttext objects

I am using gunicorn with multiple workers for my machine learning project. But the problem is when I send a train request only the worker getting the training request gets updated with the latest model after training is done. Here it is worth to…
hafiz031
  • 2,236
  • 3
  • 26
  • 48
1
vote
0 answers

Python script compiled with Pyinstaller produces pickling error due to dill.load_session()

My python script runs fine, but the executable compiled with Pyinstaller produces a PicklingError. I am using Anaconda to write my scripts, and Anaconda prompt to run the Pyinstaller.I am using Python: 3.8.5 (conda), and Platform:…
Kim Harvey
  • 11
  • 2
1
vote
0 answers

How to resolve error getting while saving xgboost model in azure databricks?

I am trying to save model in azure databricks but getting error - "It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it…
Surbhi Jain
  • 132
  • 1
  • 6