23

Official documents state that "It is not recommended to use pickle or cPickle to save a Keras model."

However, my need for pickling Keras model stems from hyperparameter optimization using sklearn's RandomizedSearchCV (or any other hyperparameter optimizers). It's essential to save the results to a file, since then the script can be executed remotely in a detached session etc.

Essentially, I want to:

trial_search = RandomizedSearchCV( estimator=keras_model, ... )
pickle.dump( trial_search, open( "trial_search.pickle", "wb" ) )
Sida Zhou
  • 3,529
  • 2
  • 33
  • 48
  • 1
    This question is also applicable to other deep learning frameworks such as Tensorflow etc. – Sida Zhou Jan 17 '18 at 07:26
  • 3
    Keras recommends to use [model.save()](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model). Scikit [recommends joblib](http://scikit-learn.org/stable/modules/model_persistence.html). After tuning the params with RandomizedSearchCV, you can just use `trial_search.best_estimator_` to get your best fitted model and then use keras recommended method on that. Why do you want to save a RandomizedSearhCV object anyways? – Vivek Kumar Jan 17 '18 at 07:37
  • 1
    Because you'd like to save the history of the search along with all the details with it, for example I'd like to know how's the 2nd best result looked like and what parameters it used, retroactively. – Sida Zhou Jan 17 '18 at 07:43
  • Then instead of the complete RandomizedSearchCV, you should look for saving the `cv_results_` attribute which will be a dict containing all the info you require. – Vivek Kumar Jan 17 '18 at 07:49
  • 1
    By definition, if you are not saving the whole object, you will be losing information. For example, on a later date, I suddenly want to compare weights of 3rd best model to the weights of 5th best model. Now instead of `pickle.load()` I must rerun those models again. Must I not? Also, I don't want to be attached to `sklearn`, since I'd like to use other hyperparam optimizers in a later date. – Sida Zhou Jan 17 '18 at 07:55
  • The RandomizedSearchCV will not save all the fitted models anyways. So if your goal is to save all the fitted models for all the combinations of your param grid over the cross-validation folds, saving RandomizedSearchCV object will not be of any use. – Vivek Kumar Jan 17 '18 at 08:00
  • I'm not 100% on this, but I'm quite sure RandomizedSearchCV object points to the estimator objects, and hence `pickle` will try to pickle those estimator objects, in this case the estimator is keras.model, and hence this question. – Sida Zhou Jan 17 '18 at 08:11
  • I dont think so. Can you if possible make an example or point me to the link which can verify when you say "I'm quite sure RandomizedSearchCV object points to the estimator objects" – Vivek Kumar Jan 17 '18 at 08:15
  • Discussion is getting far from the topic, but something like this I had in mind: https://gist.github.com/sidazhou/c26b9cf98cbbaf13ecfda8baba56c28c – Sida Zhou Jan 17 '18 at 08:30
  • Yes I agree about the topic. But what you have in mind and what you have shown on above gist is not possible for RandomizedSearchCV. I can describe how it will not save the keras models for each fold, but thats far from topic. Now since you want where "the script can be executed remotely in a detached session", I'm sure you will be only running a final version of model there (after tuning is complete). For that refer to my first comment. – Vivek Kumar Jan 17 '18 at 09:09

4 Answers4

17

As of now, Keras models are pickle-able. But we still recommend using model.save() to save model to disk.

farizrahman4u
  • 456
  • 4
  • 12
  • 1
    I have seen the recommendation in keras FAQ, but I'm wondering why is `model.save()` preferred to pickling, can you please clarify this in your answer? – S.Mohsen sh Jan 05 '19 at 08:13
  • 2
    Models saved with model.save() will be compatible with future versions of Keras and can also be exported to other platforms and implementations (deeplearning4j, Apple CoreML etc). – farizrahman4u Jan 06 '19 at 08:56
  • 8
    model.save() is buggy with python3. Every time I load the model, it predicts different things for the same input. – user2814799 Nov 29 '19 at 00:57
  • @user2814799 Is that still true!? – jtlz2 Jul 23 '21 at 12:13
  • 2
    I tried to pickle but it raised `can not pickle weakref`. Is there any to be done before pickle? – Litchy Jun 01 '22 at 02:57
  • also see [this question](https://stackoverflow.com/questions/64665776/typeerror-cant-pickle-weakref-objects-when-pickling-a-deep-learning-model) – Litchy Jun 01 '22 at 03:03
11

This works like a charm http://zachmoshe.com/2017/04/03/pickling-keras-models.html:

import types
import tempfile
import keras.models

def make_keras_picklable():
    def __getstate__(self):
        model_str = ""
        with tempfile.NamedTemporaryFile(suffix='.hdf5', delete=True) as fd:
            keras.models.save_model(self, fd.name, overwrite=True)
            model_str = fd.read()
        d = { 'model_str': model_str }
        return d

    def __setstate__(self, state):
        with tempfile.NamedTemporaryFile(suffix='.hdf5', delete=True) as fd:
            fd.write(state['model_str'])
            fd.flush()
            model = keras.models.load_model(fd.name)
        self.__dict__ = model.__dict__


    cls = keras.models.Model
    cls.__getstate__ = __getstate__
    cls.__setstate__ = __setstate__

make_keras_picklable()

PS. I had some problems, due to my model.to_json() raised TypeError('Not JSON Serializable:', obj) due to circular reference, and this error has been swallowed by the code above somehow, hence resulting in pickle function running forever.

Sida Zhou
  • 3,529
  • 2
  • 33
  • 48
  • It's not clear to me how do you use make_keras_picklable() to pickle my_model. Can you show an example pl.? – Paul.j Dec 15 '21 at 03:50
  • @Paul.j The function mutates the `keras.Model` class so that its dunder methods call the functions `__setstate__` and `__getstate__` defined in `make_keras_pickable`. Once you call that function, the get and set state dunder methods will work (sort of) with `pickle`. FWIW, it looks as though it will work with `hd5` serializations but not `SavedModel` serializations of keras, which is a pretty big limitation. – philosofool Mar 28 '23 at 21:54
11

USE get_weights AND set_weights TO SAVE AND LOAD MODEL, RESPECTIVELY.

Have a look at this link: Unable to save DataFrame to HDF5 ("object header message is too large")

#for heavy model architectures, .h5 file is unsupported.
weigh= model.get_weights();    pklfile= "D:/modelweights.pkl"
try:
    fpkl= open(pklfile, 'wb')    #Python 3     
    pickle.dump(weigh, fpkl, protocol= pickle.HIGHEST_PROTOCOL)
    fpkl.close()
except:
    fpkl= open(pklfile, 'w')    #Python 2      
    pickle.dump(weigh, fpkl, protocol= pickle.HIGHEST_PROTOCOL)
    fpkl.close()
Community
  • 1
  • 1
Anurag Gupta
  • 485
  • 5
  • 5
2

You can Pickle a Keras neural network by using the deploy-ml module which can be installed via pip 

pip install deploy-ml

Full training and deployment of a kera neural network using the deploy-ml wrapper looks like this:

import pandas as pd
from deployml.keras import NeuralNetworkBase


# load data 
train = pd.read_csv('example_data.csv')

# define the moel 
NN = NeuralNetworkBase(hidden_layers = (7, 3),
                   first_layer=len(train.keys())-1, 
                   n_classes=len(train.keys())-1)

# define data for the model 
NN.data = train

# define the column in the data you're trying to predict
NN.outcome_pointer = 'paid'

# train the model, scale means that it's using a standard 
# scaler to scale the data
NN.train(scale=True, batch_size=100)

NN.show_learning_curve()

# display the recall and precision 
NN.evaluate_outcome()

# Pickle your model
NN.deploy_model(description='Keras NN',
            author="maxwell flitton", organisation='example',
            file_name='neural.sav')

The Pickled file contains the model, the metrics from the testing, a list of variable names and their order in which they have to be inputted, the version of Keras and python used, and if a scaler is used it will also be stored in the file. Documentation is here. Loading and using the file is done by the following:

import pickle

# use pickle to load the model 
loaded_model = pickle.load(open("neural.sav", 'rb'))

# use the scaler to scale your data you want to input 
input_data = loaded_model['scaler'].transform([[1, 28, 0, 1, 30]])

# get the prediction 
loaded_model['model'].predict(input_data)[0][0]

I appreciate that the training can be a bit restrictive. Deploy-ml supports importing your own model for Sk-learn but it's still working on this support for Keras. However, I've found that you can create a deploy-ml NeuralNetworkBase object, define your own Keras neural network outside of Deploy-ml, and assign it to the deploy-ml model attribute and this works just fine:

 NN = NeuralNetworkBase(hidden_layers = (7, 3),
               first_layer=len(train.keys())-1, 
               n_classes=len(train.keys())-1)

NN.model = neural_network_you_defined_yourself
max89
  • 443
  • 5
  • 18