Learning Rate Decay in Lasagne

Question

I am using nesterov momentum for updating the weights of a Convolutional Neural Network. I am using Lasagne for building a CNN. How to implement learning rate decay for every epoch?

prediction = lasagne.layers.get_output(network)
loss = lasagne.objectives.categorical_crossentropy(prediction, target_var)
params = lasagne.layers.get_all_params(network, trainable=True)
lasagne.updates.nesterov_momentum(loss, params, learning_rate, momentum=0.9)
train_fn = theano.function([input_var_1, input_var_2, target_var], loss, updates=updates)

Can you use adam like in Keras? If so, you could just use adam or one if its variants... — pedrobisp, May 30 '16 at 17:02

score 3 · Answer 1 · answered Jun 02 '16 at 15:01

Define your learning rate as a graph element

...
learning_rate = T.scalar(name='learning_rate')
updates = lasagne.updates.nesterov_momentum(loss, params, learning_rate, momentum=0.9)
...

Add your learning rate node to train_fn

train_fn = theano.function([input_var_1, input_var_2, target_var, learning_rate], loss, updates=updates)

Now you're able to adjust the learning rate for example like this:

base_lr = 1e-4
lr_decay = 0.8

for epoch in range(epochs):
    lr = base_lr * (lr_decay ** epoch)
    loss, updates = train_fn(input_1, input_2, target, lr)

score 0 · Answer 2 · answered May 30 '16 at 22:15

I think you missing something:

prediction = lasagne.layers.get_output(network)
loss = lasagne.objectives.categorical_crossentropy(prediction, target_var)
params = lasagne.layers.get_all_params(network, trainable=True)
updates = lasagne.updates.nesterov_momentum(loss, params, learning_rate, momentum=0.9)
train_fn = theano.function([input_var_1, input_var_2, target_var], loss, updates=updates)

Learning Rate Decay in Lasagne

2 Answers2