Extremely slow model load with keras

Question

I have a set of Keras models (30) that I trained and saved using:

 model.save('model{0}.h5'.format(n_model))

When I try to load them, using load_model, the time required for each model is quite large and incremental. The loading is done as:

models = {}
for i in range(30):
    start = time.time()
    models[i] = load_model('model{0}.h5'.format(ix)) 
    end = time.time()
    print "Model {0}: seconds {1}".format(ix, end - start)

And the output is:

...
Model 9: seconds 7.38966012001
Model 10: seconds 9.99283003807
Model 11: seconds 9.7262301445
Model 12: seconds 9.17000102997
Model 13: seconds 10.1657290459
Model 14: seconds 12.5914049149
Model 15: seconds 11.652477026
Model 16: seconds 12.0126030445
Model 17: seconds 14.3402299881
Model 18: seconds 14.3761711121
...

Each model is really simple: 2 hidden layers with 10 neurons each (size ~50Kb). Why is the loading taking so much and why is the time increasing? Am I missing something (e.g. close function for the model?)

SOLUTION

I found out that to speed up the loading of the model is better to store the structure of the networks and the weights into two distinct files: The saving part:

model.save_weights('model.h5')
model_json = model.to_json()
with open('model.json', "w") as json_file:
    json_file.write(model_json)
json_file.close()

The loading part:

from keras.models import model_from_json
json_file = open("model.json", 'r')
loaded_model_json = json_file.read()
json_file.close()
model = model_from_json(loaded_model_json)
model.load_weights("model.h5")

Did you manage to solve this? I have the exact same problem with time increasing with each loaded model — Daniel, Jan 02 '18 at 05:32
@TitusPullo What is the speedup? Could you show what are the timings now? — hans, Jul 19 '19 at 09:48
I had no speed up, now load_weights works very slow. too slow, i'd say. — Alexander Sobolev, Aug 07 '19 at 21:36
if you use `with`, no need to close the file; you can remove `json_file.close()` from your saving part. — David Jones, Aug 29 '19 at 10:54

score 14 · Answer 1 · answered Mar 14 '18 at 15:09

14

I solved the problem by clearing the keras session before each load

from keras import backend as K
for i in range(...):
  K.clear_session()
  model = load_model(...)

answered Mar 14 '18 at 15:09

GearLux

149
1
3

Thank you so much for this. I have a prediction setup predicting using 1000s of models, and this saved my bacon. – user31415629 Apr 11 '18 at 09:04
1

I tried this and it seemed to corrupt the models I loaded earlier in the loop. Somehow, all models were expected to have the input that conformed to the shape of the last model I loaded. (Oh, I see @Wentai Chen mentioned it above.) – Eric McLachlan Mar 23 '19 at 09:04
@EricMcLachlan hey, can you clarify how to solve the multiple models getting corrupted. I am facing the same issue. and i dont see any comment by anyone named Wentai. I get InvalidArgumentError: Tensor dense_3/Sigmoid:0, specified in either feed_devices or fetch_devices was not found in the Graph error – Sahil Shah Apr 12 '19 at 22:18
@SahilShah: I ended up reverting my use of clear_session because it seemed to clear all previously loaded models. So, for example, I had models 0..9, but all 10 models had the implementation of model[9]. That's how it seemed at the time. – Eric McLachlan Apr 13 '19 at 13:51
Okay, but then how did you make it faster? or did you just end up letting it be slow? – Sahil Shah Apr 15 '19 at 18:47
1

As mentioned by the others, this speeds up considerably but it corrupts all models. It is therefore not a solution. We'll just have to bear the extremely slow loading. – Bert Bril Oct 14 '19 at 12:13

score 5 · Answer 2 · answered Jul 18 '18 at 11:17

I tried with K.clear_session(), and it does boost the loading time each time.
However, my models loaded in this way are not able to use model.predict function due to the following error:
ValueError: Tensor Tensor("Sigmoid_2:0", shape=(?, 17), dtype=float32) is not an element of this graph.
Github #2397 provide a detailed discussion for this. The best solution for now is to predict the data right after loading the model, instead of loading a dozens of models at the same time. After predicting each time you can use K.clear_session() to release the GPU, so that next loading won't take more time.

I tried clear_session() but I get "could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM" — Moondra, Apr 02 '19 at 20:43

score 2 · Answer 3 · answered Aug 03 '20 at 16:51

Although I'm too late to the party, on Google's Facenet (89MB), I have got some interesting results as follows,

I tried every option mentioned in the above answers, But found out that Keras in Tensorflow is slightly faster than vanilla Keras, and the results might be better on a stronger CPU.

My laptop is really old, with a configuration of 4th gen i5 (4120U), 8GB 1600MHz DDR3, and a normal SATA SSD, and I'm using Tensorflow 1.15.2 CPU version.

# Old method -- 16 to 17 seconds
from keras import backend as K
K.clear_session()
model = load_model('models/facenet_v1.h5', compile=False)
#------------------------------------------------------

# Method 1 -- 16 to 18 seconds 
from keras import backend as K
K.clear_session()
with open('models/facenet_v1.json', 'r') as j:
    json_model = j.read()

model = model_from_json(json_model)
model.load_weights('models/facenet_v1_weights.h5')
#------------------------------------------------------

# Method 2 -- 9 to 11 seconds -> Best
tf.keras.backend.clear_session()
model = tf.keras.models.load_model('models/facenet_v1.h5', compile=False)

And apart from this definitely, you'll get much better results if you have a GPU.

score 1 · Answer 4 · answered Mar 24 '20 at 09:20

1

Although it might be too late to answer this. I think this can solve the problem.

model = tf.keras.models.load_model(model_path, compile=False)

Add compile=False when you load the model.

answered Mar 24 '20 at 09:20

SAK

65
1
9

score 0 · Answer 5 · edited Jan 09 '20 at 20:50

I am having a similar problem. During training, I save the model with the lowest validation loss to a file like xyz.h5. After training completes, I load the saved model without using K.clear_session(), so it takes over a minute to load. Using K.clear_session() takes about 39 seconds to load instead. The xyz.h5 file is about 39 MB. Seems like even 39 seconds is way to long.

At any rate I decided to "go around" the delay by writing a small callback that saves the model weights for the lowest validation loss. Then I load these weights into the model to do predictions. Just include the callback in the list of callbacks when training. ie callbacks=[save_best_weights(), etc, etc]. Then after training is complete set the model weights with:

model.set_weights(save_best_weights.best_weights)

If you want to save the weights based on another metric just change the line:

current_loss=logs.get(whatever metric you choose)

in

class save_best_weights(tf.keras.callbacks.Callback):
best_weights=model.get_weights()    
def __init__(self):
    super(save_best_weights, self).__init__()
    self.best = np.Inf
def on_epoch_end(self, epoch, logs=None):
    current_loss = logs.get('val_loss')
    accuracy=logs.get('val_accuracy')* 100
    if np.less(current_loss, self.best):
        self.best = current_loss            
        save_best_weights.best_weights=model.get_weights()
        print('\nSaving weights validation loss= {0:6.4f}  validation accuracy= {1:6.3f} %\n'.format(current_loss, accuracy))

score -2 · Answer 6 · answered Mar 04 '19 at 07:24

-2

I done in this way

from keras.models import Sequential
from keras_contrib.losses import import crf_loss
from keras_contrib.metrics import crf_viterbi_accuracy

# To save model
model.save('my_model_01.hdf5')

# To load the model
custom_objects={'CRF': CRF,'crf_loss':  crf_loss,'crf_viterbi_accuracy':crf_viterbi_accuracy}

# To load a persisted model that uses the CRF layer 
model1 = load_model("/home/abc/my_model_01.hdf5", custom_objects = custom_objects)

answered Mar 04 '19 at 07:24

TRINADH NAGUBADI

377
4
9

1

Can you explain what the custom_objects are doing please? – Moondra Apr 02 '19 at 20:29
1

Please go through the below link @Moondra [https://github.com/keras-team/keras-ontrib/blob/master/keras_contrib/layers/crf.py] – TRINADH NAGUBADI Apr 04 '19 at 13:02

Extremely slow model load with keras

6 Answers6

Linked