1

I am having difficulty understanding how the training will be resumed when the model is loaded from disk when a scheduler like the one below is used.

learning_rate_scheduler = tensorflow.keras.optimizers.schedules.ExponentialDecay(
0.01, 
decay_steps=1000, 
decay_rate=0.96, 
staircase=True)

Consider this hypothetical situation, where I trained the model for one epoch and saved. Later I loaded the model and fit again. In this case, will the training resumes from the learning rate that was when the model saved previously or will it start from the pre-defined configuration of the scheduler?

Edit

I am saving my model in the standard way,

model.save("model")

Below is the optimizer config after loading. Learning rate config is same as the definition.

hour_glass_model.optimizer.get_config()

{'amsgrad': False,
 'beta_1': 0.9,
 'beta_2': 0.999,
 'decay': 0.0,
 'epsilon': 1e-07,
 'learning_rate': {'class_name': 'ExponentialDecay',
  'config': {'decay_rate': 0.96,
   'decay_steps': 1000,
   'initial_learning_rate': 0.01,
   'name': None,
   'staircase': True}},
 'name': 'Adam'}
sreagm
  • 378
  • 2
  • 10

2 Answers2

2

When you restart training after using model.save it trains with the learning rate it had when you saved the model. To make sure I wrote a simple callback using the learning rate scheduler callback.The code for the callback is shown below. I then trained a model for 5 epochs, saved the model, loaded the model and trained again. The callback prints the value of the learning rate at the beginning of each epoch and shows when training resumes the learning rate was preserved.

def scheduler(epoch, lr):
    lrin=lr
    if epoch < 2:
     lrout=lr
    else:
        lrout= lr * .5
    print ('At the start of epoch ', epoch+1, 'lr is ', lrin, ' will be set to ', lrout, ' for epoch ', epoch+2)
    return lrout
lrs=tf.keras.callbacks.LearningRateScheduler(scheduler)

put this in your code before you call model.fit. then in model.fit include

callbacks=[lrs]
Gerry P
  • 7,662
  • 3
  • 10
  • 20
1

Depends how you save your model. If you are using a standard approach with Model.save() the optimizer state is saved. the optimizer configuration is saved. The learning rate will start from the initial value

Reference: https://www.tensorflow.org/guide/keras/save_and_serialize

Please also see save API which has by default include_optimizer=True

Proko
  • 1,871
  • 1
  • 11
  • 23
  • I am saving the model as SavedModel. The loaded model's optimizer includes the scheduler with its default config. So i am doubtful about whether the training starts again from the initial learning rate or the learning rate when the training was previously stopped when saving. – sreagm Mar 01 '21 at 10:32
  • @sreagm please update your question with code snipped that exactly matches your way of saving the model. Only then I can tell you for sure – Proko Mar 01 '21 at 10:44
  • Thanks, @Proko for the valuable pointers. – sreagm Mar 02 '21 at 06:13