10

I am loading a model in keras with model.load() and am finding that the first prediction is taking more than 10x longer to calculate than follow on predictions, any ideas why this could be occurring or suggestions to make the load-initialise-first prediction cycle speed up would be greatly appreciated.

I am using Tensorflow backend with CPU processing.

Thanks for the help, Denym

Shai
  • 111,146
  • 38
  • 238
  • 371
Denym
  • 181
  • 1
  • 8
  • Are you sure it is the first prediction taking so much time and not the loading? What do you mean by 'first prediction'? Are you running `model.predict()` multiple times? – McLawrence Aug 31 '17 at 08:51
  • I am loading the saved model with the model.load(), then building the data set and running model.predict(), then the dataset is modified and another prediction run, then dataset is modified and another prediction is run etc. the model.load appears to complete quite quickly, then the first prediction takes roughly 10x the amount of time to produce a prediction as each follow on prediction. – Denym Aug 31 '17 at 13:24
  • How do you modify the dataset? Are all datasets on which you predict of the same size? – McLawrence Aug 31 '17 at 13:26
  • yes, it is another dataset of the same size – Denym Aug 31 '17 at 14:08
  • Could you provide a code you use in this case? – Marcin Możejko Aug 31 '17 at 14:47
  • not really, it is simply feeding dataset1 into the model that was loaded using model.load() and then it is fed additional data sets, the first prediction takes about 8 seconds with each subsequent prediction with new datasets taking sub 1 second – Denym Aug 31 '17 at 15:01
  • I am facing the same problem. As you have mentioned I loaded the model in json format. It still takes more time during first prediction. – AKSHAYAA VAIDYANATHAN Dec 07 '17 at 14:20

1 Answers1

7

Ok so I have found the answer that works for me:

if you are loading many models simultaneously don't use the keras model.load function, save your structure as a json/yaml and the weights as a .h5 and load as per the keras examples.

the model.load function is much quicker when dealing with less than 5 models however load times exponentially increase the more models you simultaneously load.

loading from json and weights from .h5 was 10x faster when loading 100 models simultaneously, and while there is some slow down per model when loading structure and weights method it is linear rather than exponential, making it significantly faster when loading many models at once.

Denym
  • 181
  • 1
  • 8
  • 2
    You are talking here about model load time , but first you were complaining about the .predict time ...I am asking because in my case with single model, The first call to model.predict takes ~5 sec then other calls to predict takes about 0.9 sec...I am wondering if you did found solution for this ? – Stav Bodik Feb 05 '18 at 12:50
  • 3
    Hi Stav, the best solution I found for this was to initialise the model as an object and have it make a prediction on null values on initialisation. Sorry for the late reply and I hope this is helpful. If you have found a better way to speed up the first prediction I would love to hear about it too. – Denym Apr 02 '18 at 23:59
  • Thanks you very much ! – Stav Bodik Apr 08 '18 at 07:39
  • This is more related to first time model loading, not the first prediction time. Assumed that the model is loaded already, the prediction time must be calculated as time spent calling `model.predict`• and load `load_model`. – loretoparisi Mar 20 '19 at 10:19
  • Yip. Your suggestion has resulted in my 45+ classifiers loading MUCH faster. Thanks a lot. – Eric McLachlan Jul 29 '19 at 11:13
  • No worries, glad it helped – Denym Jul 30 '19 at 12:09