Stack Overflow
Stack Overflow

Questions tagged [pickle]

Pickle is an object serialization module for Python. Use this tag together with the Python tag for questions related to storing or loading objects with Pickle.

Pickle provides a powerful set of tools for serializing and de-serializing Python objects.

4936 questions
66
votes
4 answers

Python: Can't pickle type X, attribute lookup failed

I am trying to pickle a namedtuple: from collections import namedtuple import cPickle class Foo: Bar = namedtuple('Bar', ['x', 'y']) def baz(self): s = set() s.add(Foo.Bar(x=2, y=3)) print cPickle.dumps(s) if…
Nick Heiner
  • 119,074
  • 188
  • 476
  • 699
64
votes
4 answers

What is the difference between rb and r+b modes in file objects

I am using pickle module in Python and trying different file IO modes: # works on windows.. "rb" with open(pickle_f, 'rb') as fhand: obj = pickle.load(fhand) # works on linux.. "r" with open(pickle_f, 'r') as fhand: obj =…
Iamcool
  • 1,387
  • 2
  • 12
  • 24
61
votes
3 answers

_pickle in python3 doesn't work for large data saving

I am trying to apply _pickle to save data onto disk. But when calling _pickle.dump, I got an error OverflowError: cannot serialize a bytes object larger than 4 GiB Is this a hard limitation to use _pickle? (cPickle for python2)
Jake0x32
  • 1,402
  • 2
  • 11
  • 18
59
votes
10 answers

Can't pickle defaultdict

I have a defaultdict that looks like this: dict1 = defaultdict(lambda: defaultdict(int)) The problem is, I can't pickle it using cPickle. One of the solution that I found here is to use module-level function instead of a lambda. My question is,…
Fynn Mahoney
  • 699
  • 1
  • 7
  • 10
58
votes
4 answers

What can multiprocessing and dill do together?

I would like to use the multiprocessing library in Python. Sadly multiprocessing uses pickle which doesn't support functions with closures, lambdas, or functions in __main__. All three of these are important to me In [1]: import pickle In [2]:…
MRocklin
  • 55,641
  • 23
  • 163
  • 235
55
votes
3 answers

Convert numpy array type and values from Float64 to Float32

I am trying to convert threshold array(pickle file of isolation forest from scikit learn) of type from Float64 to Float32 for i in range(len(tree.tree_.threshold)): tree.tree_.threshold[i] = tree.tree_.threshold[i].astype(np.float32) ​ Then…
Akshay Tilekar
  • 1,910
  • 2
  • 12
  • 22
54
votes
2 answers

Python2: Should I use Pickle or cPickle?

Python 2 has both the pickle and cPickle modules for serialization. cPickle has an obvious advantage over pickle: speed. What, if any, advantage does pickle have over cPickle?
user200783
  • 13,722
  • 12
  • 69
  • 135
51
votes
2 answers

Can't find module cPickle using Python 3.5 and Anaconda

I am trying to use cPickle on a windows box, using Anaconda. I am using python 3.5. I am not using a virtualenv (though probably should be). When I try to import cPickle I get "ImportError: No module named 'cPickle'" Python 3.5.0 |Anaconda custom…
Tom Walker
  • 837
  • 1
  • 8
  • 12
50
votes
3 answers

multiprocessing.Pool - PicklingError: Can't pickle : attribute lookup thread.lock failed

multiprocessing.Pool is driving me crazy... I want to upgrade many packages, and for every one of them I have to check whether there is a greater version or not. This is done by the check_one function. The main code is in the Updater.update method:…
rubik
  • 8,814
  • 9
  • 58
  • 88
49
votes
6 answers

Save a dictionary to a file (alternative to pickle) in Python?

Answered I ended up going with pickle at the end anyway Ok so with some advice on another question I asked I was told to use pickle to save a dictionary to a file. The dictionary that I was trying to save to the file was members = {'Starspy' :…
wakey
  • 2,283
  • 4
  • 31
  • 58
49
votes
10 answers

ValueError: insecure string pickle

When I am trying to load something I dumped using cPickle, I get the error message: ValueError: insecure string pickle Both the dumping and loading work are done on the same computer, thus same OS: Ubuntu 8.04. How could I solve this problem?
Peter Long
  • 491
  • 1
  • 4
  • 4
48
votes
3 answers

Why can I pass an instance method to multiprocessing.Process, but not a multiprocessing.Pool?

I am trying to write an application that applies a function concurrently with a multiprocessing.Pool. I would like this function to be an instance method (so I can define it differently in different subclasses). This doesn't seem to be possible; as…
dpitch40
  • 2,621
  • 7
  • 31
  • 44
47
votes
7 answers

Not able to pip install pickle in python 3.6

I am trying to run the following code: import bs4 as bs import pickle import requests import lxml def save_sp500_tickers(): resp = requests.get("https://en.wikipedia.org/wiki/List_of_S%26P_500_companies") soup = bs.BeautifulSoup(resp.text,…
satyaki
  • 581
  • 2
  • 6
  • 14
46
votes
2 answers

Why can't generators be pickled?

Python's pickle (I'm talking standard Python 2.5/2.6/2.7 here) cannot pickle locks, file objects etc. It also cannot pickle generators and lambda expressions (or any other anonymous code), because the pickle really only stores name references. In…
Radim
  • 4,208
  • 3
  • 27
  • 38
46
votes
3 answers

How to Reduce the time taken to load a pickle file in python

I have created a dictionary in python and dumped into pickle. Its size went to 300MB. Now, I want to load the same pickle. output = open('myfile.pkl', 'rb') mydict = pickle.load(output) Loading this pickle takes around 15 seconds. How can I reduce…
iNikkz
  • 3,729
  • 5
  • 29
  • 59
1 2 3
…
99 100