Get speed of model evaluation with Keras

Question

I would like to retrieve metrics of speed of the model on validation data, in order to compare different parameters and their impact on speed. For example, the time spent per batch on validation data. Or, since I use hyperopt, the time spent per iteration/trial, also on validation data.

Is there any way to do so with the outputs of fit() with validation_split>0, predict() or evaluate() or the attributes of Trials in hyperopt?

If not, I guess I'll have to put time landmarks in the code but it would not be ideal for me.

Thank you!

score 0 · Answer 1 · answered Jul 28 '19 at 18:17

you can achieve this by logging batch times in a custom callback invoked at the end of each evaluation batch. It would look something like this:

from keras.callbacks import Callback
import time

class BatchTimeCallback(Callback):
    def on_train_begin(self, logs={}):
        self.batch_times = []

    def on_batch_end(self, batch, logs={}):
        self.batch_times.append(time.time())

batch_time_callback = BatchTimeCallback()

...

model.evaluate(..., callbacks=[batch_time_callback])

print(batch_time_callback.batch_times)

Please note that is isn't supported in the latest release (2.2.4), so you have to use the master branch.

Get speed of model evaluation with Keras

1 Answers1