91

I'm trying to restore a pickled config file from RLLib (json didn't work as shown in this post), and getting the following error:

config = pickle.load(open(f"{path}/params.pkl", "rb"))

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-c964561b863c> in <module>
----> 1 config = pickle.load(open(f"{path}/params.pkl", "rb"))

ValueError: unsupported pickle protocol: 5

Python Version = 3.7.0

How can I open this file in 3.7?

hubbs5
  • 1,235
  • 1
  • 12
  • 22

4 Answers4

117

For pandas users who saved a dataframe to a pickle file with protocol 5 in python 3.8 and need to load it into python 3.6 which only supports protocol 4 (I'm looking at you google colab):

!pip3 install pickle5
import pickle5 as pickle
with open(path_to_protocol5, "rb") as fh:
  data = pickle.load(fh)

Could also save into a protocol-4 pickle from python 3.6

data.to_pickle(path_to_protocol4)

Update: If facing this when loading a model from stable-baselines3:

!pip install --upgrade --quiet cloudpickle pickle5
from stable_baselines3 import PPO
# restart kernel if in jupyter notebook

# Might not need this dict in all cases
custom_objects = {
    "lr_schedule": lambda x: .003,
    "clip_range": lambda x: .02
}
model = PPO.load("path/to/model.zip", custom_objects=custom_objects)

Tested on 2021-05-31 with env:

cloudpickle: 1.6.0
pickle5: 0.0.11 
stable-baselines3: 1.0

Reference: https://brainsteam.co.uk/2021/01/14/pickle-5-madness-with-mlflow/

Shadi
  • 9,742
  • 4
  • 43
  • 65
65

Use pickle5 or load it into python 3.8+ and then serialize it to a lower version of it using the protocol parameter.

hd1
  • 33,938
  • 5
  • 80
  • 91
  • 3
    Didn't know about pickle5! I was in the process of setting up a 3.8 virtual env and installing, all the dependencies, but this is much easier. Thanks! – hubbs5 Aug 09 '20 at 18:23
  • 1
    I just found it myself. PyPi needs to be better at advertising itself. – hd1 Aug 09 '20 at 19:24
  • 2
    ModuleNotFoundError: No module named 'pickle5' though I import it. – Fatemeh Asgarinejad Jan 13 '21 at 05:49
  • 1
    Is it installed in your venv? – hd1 Jan 13 '21 at 06:03
  • 4
    A concrete example of serializing to a lower version with pandas data frame: df.to_pickle('filename.pkl', protocol=4) – Nir Feb 25 '21 at 00:18
  • Is there a way to save a pickle for backward compatibility? It seems impossible. I save using data.to_pickle('data.pkl_p4',protocol=4) but it was impossible to read (ModuleNotFoundError: No module named 'pandas.core.internals.managers'; 'pandas.core.internals' is not a package) – Ofer Rahat Sep 07 '22 at 08:46
  • Do you have that package -- pandas.core.internals -- installed with the same revision as was used to generate the pickle? – hd1 Sep 07 '22 at 15:54
0

In the event that you cannot load pickle5 because of its dependencies (mainly Visual Building c++), another solution could be that you change the Python interpreter you're using (to the old one, before the error occured). For me, I was getting this error after I ran a program in IDLE that I had been running in Spyder. When I ran it again within Spyder, it dropped this error.

Python Error: Unsupported Pickle Protocol 5

To resolve this, within Spyder I changed my Python interpreter to the Python I was using with IDLE (Tools -> Preferences). Once I rebooted Spyder, I had to install the necessary dependencies with command prompt so that within Spyder the console could be used:

pip install spyder-kernels

Naturally, this may introduce some irregularities within Spyder (namely, packages it's supposed to come with are no longer there because of the different interpreter). These should easily be sussed out when debugging, and resolved using standard pip installs.

Once you recover your (thought to be lost) files, it might be wise to think about reverting back to Spyder's Python interpreter, and updating code to elegantly handle this problem (I would love if somebody could suggest in comments how to do this that didn't require pickle5!)

ntk4
  • 1,247
  • 1
  • 13
  • 18
0

If this error is due to heroku deployment then check your python version of local setup and heroku setup. If both are different then it might lead you to this error.
Solution:

  • Create a runtime.txt file in your application's base directory

Then inside your runtime.txt mention your local python version so that there are no conflicts.

python-3.9.2
sauravjoshi23
  • 837
  • 11
  • 9