How can I compute the hessian of the loss function in MXNet?

Question

I'm learning MXNet at the moment and I'm working on a problem using neural nets. I'm interested in observing the curvature of my loss function with respect to the network weights but as best I can tell higher order gradients are not supported for neural network functions at the moment. Is there any (possibly hacky) way that I could still do this?

score 1 · Answer 1 · answered Jul 09 '18 at 18:59

You can follow the discussion here

The gist of it is that not all operators support higher order gradients at the moment.

In Gluon you can try the following:

with mx.autograd.record():
  output = net(x)
  loss = loss_func(output)
  dz = mx.autograd.grad(loss, [z], create_graph=True)  # where [z] is the parameter(s) you want

dz[0].backward()  # now the actual parameters should have second order gradients

Taken from this forum thread

How can I compute the hessian of the loss function in MXNet?

1 Answers1