3

I'm currently working on recurrent neural nets using Lasagne / Theano.

While training, updates are calculated using Theano's symbolic gradient.

grads = theano.grad(loss_or_grads, params)

While the gradient expression is perfectly fine in general, I'm also interested in the gradient values in order to monitor training.

My question now is if there is a built-in method to also get gradient values, which I haven't found so far, or if I'll have to do it myself.

Thanks in advance

s1hofmann
  • 1,491
  • 1
  • 11
  • 22

1 Answers1

5

I'm not aware of any lasagne function to evaluate the gradient, but you can get it yourself with simple theano function.

Say we have the following theano variables:

  • inputs = Inputs to the network
  • targets = Target outputs of the network
  • loss = Value of the loss function, defined as a function of network outputs and targets
  • l_hid = Recurrent layer of the network, type lasagne.layers.RecurrentLayer

Say we're interested in the gradient of the loss function w.r.t. the recurrent weights:

grad = theano.grad(loss, l_hid.W_hid_to_hid)

Define a theano function to get a numerical value for the gradient

get_grad = theano.function([inputs, targets], grad)

Now, just call get_grad for any value of the inputs and targets (e.g. the current minibatch). get_grad() doesn't need to be passed the value of the weights because they're stored as a theano shared variable.

user20160
  • 1,396
  • 7
  • 10