tf.hessians(f, x): not the same value as tf.gradients(tf.gradients(f, x), x) for an isolated, single variable?

Question

So I have a tf.Variable() named W that is of shape=[1].

I get that the tf.hessians(loss, W) isn't equal to the tf.gradients(tf.gradients(loss, W), W), despite it should be the same thing: a second order derivative.

Here is a small gist for the code where I get the issue if I try to replace the Hessians by the double gradients: https://gist.github.com/guillaume-chevalier/6b01c4e43a123abf8db69fa97532993f Yet I observe that results do not converge if I use double gradients.

Answer 1

Double gradient is called Laplacian: Laplace Operator. It is taking derivative w.r.t the same x variable twice. However, Hessian matrix is taking derivative w.r.t combination of two x variables: Hessian Matrix. Thus the trace of Hessian would be Laplacian, which is double gradient.