Keras: how to output learning rate onto tensorboard

Question

I added a callback to decay the learning rate:

 keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=100, 
                                   verbose=0, mode='auto',epsilon=0.00002, cooldown=20, min_lr=0)

Here is my tensorboard callback:

keras.callbacks.TensorBoard(log_dir='./graph/rank{}'.format(hvd.rank()), histogram_freq=10, batch_size=FLAGS.batch_size,
                            write_graph=True, write_grads=True, write_images=False)

I want to make sure the learning rate scheduler has kicked in during training, so I want to output the learning rate onto tensorboard. But I can not find where I can set it.

I also checked the optimizer api, but no luck.

keras.optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)

How can I output the learning rate to tensorboad?

score 35 · Accepted Answer · edited Aug 06 '21 at 10:01

35

According to the author of Keras, the proper way is to subclass the TensorBoard callback:

from keras import backend as K
from keras.callbacks import TensorBoard

class LRTensorBoard(TensorBoard):
    # add other arguments to __init__ if you need
    def __init__(self, log_dir, **kwargs):
        super().__init__(log_dir=log_dir, **kwargs)

    def on_epoch_end(self, epoch, logs=None):
        logs = logs or {}
        logs.update({'lr': K.eval(self.model.optimizer.lr)})
        super().on_epoch_end(epoch, logs)

Then pass it as part of the callbacks argument to model.fit (credit Finncent Price):

model.fit(x=..., y=..., callbacks=[LRTensorBoard(log_dir="/tmp/tb_log")])

edited Aug 06 '21 at 10:01

Bremsstrahlung

686
1
9
23

answered Apr 26 '18 at 10:06

alkamid

6,970
4
28
39

Note that @alkamid's answer is for python 3, if you are in python 2, you'll need to pass the CHILD class name and the current instance to super. For this particular example `super()` -> `super(LRTensorBoard,self)` works. Answer explaining this syntactic difference can be found here: https://stackoverflow.com/questions/30633889/typeerror-super-does-not-take-key-word-arguments – Finncent Price Feb 20 '19 at 14:43
2

Instructions for how to use this callback in Keras inside the fit method of your model are as follows. Supply a list of callbacks to the callbacks variable like so: `model.fit(x=something,y=something,callbacks=[LRTensorboard(log_dir='path_to_log_dir')])` – Finncent Price Feb 20 '19 at 14:47
`logs = logs or {}; logs.update(lr=K.eval(self.model.optimizer.lr))` this is better, cause logs, could be None – Khan Aug 26 '19 at 16:43
1

I would add ` def __init__(self, **kwargs): # add other arguments to __init__ if you need super().__init__(**kwargs) ` for the function not to block other arguments to tensorboard. – Alon Samuel Sep 24 '19 at 13:07
1

@Khan I'm not sure where the `logs=None` convention is coming from, but Keras/TensorFlow tutorials seem to be using it. – alkamid Sep 27 '19 at 14:09
This is way better than what [the official docs suggest](https://www.tensorflow.org/tensorboard/scalars_and_keras#logging_custom_scalars). If I would follow that and would want to play with different decays, I'd have to write own impls of each scheduler. – hoefling Nov 22 '20 at 22:35
Although the approach seems legit; I get the following error before the training start ```AlreadyExistsError? another profiler already in use``` – NikSp Dec 27 '20 at 21:11

score 12 · Answer 2 · answered Nov 27 '20 at 22:03

Note that with the current nightly version of tf (2.5 - probably earlier) learning rates using LearningRateSchedule are automatically added to tensorboard's logs. The following solution is only necessary if you're adapting the learning rate some other way - e.g. via ReduceLROnPlateau or LearningRateScheduler (different to LearningRateSchedule) callbacks.

While extending tf.keras.callbacks.TensorBoard is a viable option, I prefer composition over subclassing.

class LearningRateLogger(tf.keras.callbacks.Callback):
    def __init__(self):
        super().__init__()
        self._supports_tf_logs = True

    def on_epoch_end(self, epoch, logs=None):
        if logs is None or "learning_rate" in logs:
            return
        logs["learning_rate"] = self.model.optimizer.lr

This allows us to compose multiple similar callbacks, and use the logged learning rate in multiple other callbacks (e.g. if you add a CSVLogger it should also write the learning rate values to file).

Then in model.fit

model.fit(
    callbacks=[
        LearningRateLogger(),
        # other callbacks that update `logs`
        tf.keras.callbacks.TensorBoard(path),
        # other callbacks that use updated logs, e.g. CSVLogger
    ],
    **kwargs
)

score 2 · Answer 3 · answered Mar 07 '18 at 12:44

You gave the optimizer's code twice, instead of TensorBoard Callback. Anyway, I didn`t find the way to display the learning rate on TensorBoard. I am plotting it after the training finished, taking data from History object:

nb_epoch = len(history1.history['loss'])
learning_rate=history1.history['lr']
xc=range(nb_epoch)
plt.figure(3,figsize=(7,5))
plt.plot(xc,learning_rate)
plt.xlabel('num of Epochs')
plt.ylabel('learning rate')
plt.title('Learning rate')
plt.grid(True)
plt.style.use(['seaborn-ticks'])

The chart looks like this: LR plot

Sorry, that is not exactly what you are asking about, but perhaps could help.

sorry for the error. your solution is good, but don't work for me if I want monitor a training process that will take a long time. — scott huang, Mar 08 '18 at 01:57

score 1 · Answer 4 · answered Nov 03 '18 at 03:04

class XTensorBoard(TensorBoard):
    def on_epoch_begin(self, epoch, logs=None):
        # get values
        lr = float(K.get_value(self.model.optimizer.lr))
        decay = float(K.get_value(self.model.optimizer.decay))
        # computer lr
        lr = lr * (1. / (1 + decay * epoch))
        K.set_value(self.model.optimizer.lr, lr)

    def on_epoch_end(self, epoch, logs=None):
        logs = logs or {}
        logs['lr'] = K.get_value(self.model.optimizer.lr)
        super().on_epoch_end(epoch, logs)

callbacks_list = [XTensorBoard('./logs')]
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=20, batch_size=32, verbose=2, callbacks=callbacks_list)

lr curve in tensorboard

Code only answers are really discouraged. Please provide explanation what you are doing too! — itsmysterybox, Nov 03 '18 at 03:21

score 0 · Answer 5 · answered Feb 08 '22 at 11:39

For tensorflor 2.5 if you have some custom Learning Rate Scheduler:

class LearningRateLogger(tf.keras.callbacks.Callback):
def __init__(self, log_dir):
    super().__init__()
    self._supports_tf_logs = True
    self.log_dir = log_dir
    
def set_model(self, model):                                                                                                                                                                                                                                       
    self.model = model                                                                                                                                                                                                                                            
    self.sess = tf.compat.v1.keras.backend.get_session()
    self.writer = tf.summary.create_file_writer(self.log_dir)

def on_epoch_end(self, epoch, logs=None):
    if logs is None or "learning_rate" in logs:
        return
    logs["learning_rate"] = self.model.optimizer.lr
    logs.update({'learning_rate': self.model.optimizer.lr})
    self._write_logs(logs, epoch)
    
def _write_logs(self, logs, index):

    with self.writer.as_default():                                                                                                                                                                                                                               
        for name, value in logs.items():                                                                                                                                                                                                                              
            if name in ['batch', 'size']:                                                                                                                                                                                                                             
                continue                                                                                                                                                                                                                    
            if isinstance(value, np.ndarray):
                tf.summary.scalar(name, value.item(), step=index)                                                                                                                                                                                                             
            else:
                tf.summary.scalar(name, value, step=index)
    
        self.writer.flush()

Then for calling the callback in your model.fit:

model.fit(x=..., y=..., callbacks=[LearningRateLogger(log_dir="/path/to/folder/where/tensorboard/is/logging")])

score 0 · Answer 6 · answered Aug 27 '23 at 01:57

0

Just make sure the ReduceLROnPlateau callback comes before the TensorBoard callback in the list of callbacks. Then you will see an lr metric logged to TensorBoard every epoch.

answered Aug 27 '23 at 01:57

matangover

357
2
12

Keras: how to output learning rate onto tensorboard

6 Answers6

Linked