What is the difference between step size and learning rate in machine learning?

Question

I am using TensorFlow to implement some basic ML code. I was wondering if anyone could give me a short explanation of the meaning of and difference between step size and learning rate in the following functions.

I used tf.train.GradientDescentOptimizer() to set the parameter learning rate and linear_regressor.train() to set the number of steps. I've been looking through the documentation on tensorflow.org for these functions but I still do not have a complete grasp of the meaning of these parameters.

Thank you and let me know if there is any more info I can provide.

Ash · Accepted Answer · 2020-01-25T11:59:48.617

3

In SGD, you compute the gradient for a batch and move the parameters in the direction of said gradient by an amount defined by the learning rate lr:

params=old_params - lr* grad

where grad is the gradient of the loss w.r.t the params.

The step in tensorflow or similar libraries usually just denotes the number of such updates per epoch. So if you have step=1000 and lr=0.5, you will be calling the pseudocode above 1000 times with lr=0.5 in each epoch.

edited Jan 25 '20 at 11:59

answered Jan 25 '20 at 11:55

Ash

4,611
6
27
41

1

Yes, but that 1000 steps would be in one epoch. So, if 10 epochs, the number of updates would be 10000 – ggaurav Jan 25 '20 at 11:57
@ggaurav Thanks, you're right. I will update the answer. – Ash Jan 25 '20 at 11:58

What is the difference between step size and learning rate in machine learning?

1 Answers1