ReduceLRonPlateau keeps decreasing LR across multiple models

Asked Jan 12 '23 at 16:21

Active Jan 12 '23 at 16:37

Viewed 104 times

I'm using ReduceLROnPlateau for multiple experiments, but I'm getting lower and lower initial learning rate for each conjsecutive modewl run.

from tensorflow.keras.callbacks import ReduceLROnPlateau


for model in models:
    reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=10)
    model.fit(dataset, epochs=10, validation_data=val_dataset, callbacks=reduce_lr)

The learning rate on the output log looks as follows:

Model #1
  Epoch 1  ... lr: 0.01
  ...
  Epoch 21 ... lr: 0.005

Model #2
  Epoch 1  ... lr: 0.005
  ...
  Epoch 25 ... lr: 0.001

and so on. (don't look at the numbers I've siplified the output)

How do I tell the model or to the callback to start from the same learning rate each time?

edited Jan 12 '23 at 16:37

asked Jan 12 '23 at 16:21

Mateusz Dorobek

This state is saved on the callback object, so the obvious solution is to use a different callback instance for each training run. – Dr. Snoopy Jan 12 '23 at 16:34
Actually I'm creating a new instance each iteration (I've fixed the snippet) – Mateusz Dorobek Jan 12 '23 at 16:38
1

You are probably using the same optimizer instance for all models. – xdurch0 Jan 12 '23 at 17:39
Yep, I was using the same model, all over again, but your answer helped to find that. Thanks. – Mateusz Dorobek Jan 12 '23 at 18:34

ReduceLRonPlateau keeps decreasing LR across multiple models

0 Answers0