Plotting learning rate vs Loss

Question

I am trying to find the best learning rate by multiplying the learning rate by a constant factor and them training the model on the the varying learning rates .I need to choose the learning rate at the turning point where the loss starts to increase again. To do this I need to visualize the learning rate vs loss plot. How do I do this.

Method for varying rate is

import math
l_rates = []
def schedule(epoch , lr):
  lr_new = lr * math.exp(math.log10(math.pow(10,6))/500)
  l_rates.append(lr_new)
  return lr_new

lr_scheduler_cb = keras.callbacks.LearningRateScheduler(schedule)

learning_rate_history = model1.fit(train_x , train_y , epochs=500 ,
                     callbacks=[lr_scheduler_cb])

score 0 · Answer 1 · answered Sep 04 '21 at 17:31

The way you formulated your questions suggests you want to try different models with constant learning rates to find the best, but you're increasing your learning rate on the same model. This is very likely a bad idea for multiple reasons.

You're not testing the model with different learning rates, you test what happens if you increase the learning rate each epoch.
It's common in supervised learning to decrease the learning rate as you get close to the optimum and need to take smaller steps, not bigger ones.

To the actual plotting, I suggest using matplotlib:

import numpy as np
import matplotlib.pyplot as plt

# example values
l_rates = np.array([1e-5, 1e-4, 1e-3, 1e-2, 1e-1])
learning_rate_history = np.random.random(size=5)

plt.plot(l_rates, learning_rate_history)
plt.show();

On a side note, you're doing grid search while random search is better.

trying to test the same model with increasing learning rate ,then plot the loss vs learning rate and choose the learning rate where loss starts to increase again. (learning rate varies from 1e-5 to 10 — Adarsh Singh, Sep 05 '21 at 06:20

Plotting learning rate vs Loss

1 Answers1