Save history of model.fit for different epochs

Question

I was training my model with epoch=10. I again retrained with epoch=3. and again epoch 5. so for every time i train model with epoch=10, 3, 5. I want to combine the history of all the 3. FOr example, let h1 = history of model.fit for epoch=10, h2 = history of model.fit for epoch=3, h3 = history of model.fit for epoch=5.

Now in variable h, i want h1 + h2 + h3. All history to be appended to single variable so that i can plot some graphs.

the code is,

start_time = time.time()

model.fit(x=X_train, y=y_train, batch_size=32, epochs=10, validation_data=(X_val, y_val), callbacks=[tensorboard, checkpoint])

end_time = time.time()
execution_time = (end_time - start_time)
print(f"Elapsed time: {hms_string(execution_time)}")


start_time = time.time()

model.fit(x=X_train, y=y_train, batch_size=32, epochs=3, validation_data=(X_val, y_val), callbacks=[tensorboard, checkpoint])

end_time = time.time()
execution_time = (end_time - start_time)
print(f"Elapsed time: {hms_string(execution_time)}")

start_time = time.time()

model.fit(x=X_train, y=y_train, batch_size=32, epochs=5, validation_data=(X_val, y_val), callbacks=[tensorboard, checkpoint])

end_time = time.time()
execution_time = (end_time - start_time)
print(f"Elapsed time: {hms_string(execution_time)}")

score 8 · Accepted Answer · answered Mar 18 '20 at 01:04

You can achieve this functionality by creating a class which sub-classes tf.keras.callbacks.Callback and use the object of that class as callback to model.fit.

import csv
import tensorflow.keras.backend as K
from tensorflow import keras
import os

model_directory='./xyz' # directory to save model history after every epoch 

class StoreModelHistory(keras.callbacks.Callback):

  def on_epoch_end(self,batch,logs=None):
    if ('lr' not in logs.keys()):
      logs.setdefault('lr',0)
      logs['lr'] = K.get_value(self.model.optimizer.lr)

    if not ('model_history.csv' in os.listdir(model_directory)):
      with open(model_directory+'model_history.csv','a') as f:
        y=csv.DictWriter(f,logs.keys())
        y.writeheader()

    with open(model_directory+'model_history.csv','a') as f:
      y=csv.DictWriter(f,logs.keys())
      y.writerow(logs)


model.fit(...,callbacks=[StoreModelHistory()])

Then you can load the csv file and plot model's loss, learning rate, metrics, etc.

import pandas as pd
import matplotlib.pyplot as plt

EPOCH = 10 # number of epochs the model has trained for

history_dataframe = pd.read_csv(model_directory+'model_history.csv',sep=',')


# Plot training & validation loss values
plt.style.use("ggplot")
plt.plot(range(1,EPOCH+1),
         history_dataframe['loss'])
plt.plot(range(1,EPOCH+1),
         history_dataframe['val_loss'],
         linestyle='--')
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()

score 4 · Answer 2 · answered Mar 17 '20 at 21:34

Every time you call model.fit(), it returns a keras.callbacks.History object whose history attribute contains a dictionary. The keys to the dictionary are loss for training, val_loss for validation loss, and any other metrics that you might have set while compiling.

Therefore, in your case, you could do:

hist1 = model.fit(...)

# other code lines

hist2 = model.fit(...)

# other code lines

hist3 = model.fit(...)

# create an empty dict to save all three history dicts into
total_history_dict = dict()

for some_key in hist1.keys():
    current_values = [] # to save values from all three hist dicts
    for hist_dict in [hist1.history, hist2.history, hist3.history]:
        current_values += hist_dict[some_key]
    total_history_dict[some_key] = current_values

Now, total_history_dict is a dictionary keys of which are as usual loss, val_loss, other metrics and values lists showing loss/metrics for each epoch. (Length of the list would be sum of number of epochs in all three calls to model.fit)

You could now use the dictionary to plot things using matplotlib or save it to a pandas dataframe, etc...

score 1 · Answer 3 · answered Apr 25 '21 at 21:03

In 2020 you can use build in CSVLogger with append=True

Save example:

epoch,accuracy,loss,val_accuracy,val_loss
0,0.7649424076080322,0.49990198016166687,0.6675007939338684,0.8114446401596069
1,0.8209356665611267,0.406406044960022,0.7569596767425537,0.5224416851997375

Riley K · Answer 4 · 2022-08-07T17:04:25.880

Most answers here (including the accepted one) overcomplicate the task of combining the training log data. Note this method is not effective when dealing with many metrics. If this is the case, it is better to use other answers to combine history objects. Tensorflow's recommended way to merge multiple training log data (https://www.tensorflow.org/tutorials/images/transfer_learning#continue_training_the_model) is as follows:

Extract the initial training log data points. Here I include accuracy and validation accuracy.

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

Extract training log data points from successive training runs. Then append them to the prior training metrics.

acc += history2.history['accuracy']
val_acc += history2.history['val_accuracy']

loss += history2.history['loss']
val_loss += history2.history['val_loss']

Graph results

plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.ylim([0.8, 1])
plt.plot([initial_epochs-1,initial_epochs-1],
          plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.ylim([0, 1.0])
plt.plot([initial_epochs-1,initial_epochs-1],
         plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()

Save history of model.fit for different epochs

4 Answers4