2

Second partial derivative test is a simple way to tell whether a critical point is a minimum, a maximum, or a saddle. I am currently toying with the idea of implementing such test for a simple neural network in tensorflow. The following set of weights is used for modeling an XOR neural network with 2 inputs, 1 hidden layer with 2 hidden units, and 1 output unit:

weights = {
    'h1': tf.Variable(np.empty([2, 2]), name="h1", dtype=tf.float64),
    'b1': tf.Variable(np.empty([2]), name="b1", dtype=tf.float64),
    'h2': tf.Variable(np.empty([2, 1]), name="h2", dtype=tf.float64),
    'b2': tf.Variable(np.empty([1]), name="b2", dtype=tf.float64)
}

Both the gradients and the hessians can now be obtained as follows:

gradients = tf.gradients(mse_op, [weights['h1'], weights['b1'], weights['h2'], weights['b2']])
hessians = tf.hessians(mse_op, [weights['h1'], weights['b1'], weights['h2'], weights['b2']])

Where mse_op is the MSE error of the network.

Both gradients and hessians compute just fine. The dimensionality of the gradients is equal to the dimensionality of the original inputs. The dimensionality of the hessians obviously differs.

The question: is it a good idea, and is it even possible to conveniently compute the eigenvalues of the hessians generated by tf.hessian applied to the given set of weights? Will the eigenvalues be representative of what I think they represent - i.e., will I be able to say that if overall, both positive and negative values are present, then we can conclude that the point is a saddle point?

So far, I have tried the following out-of-the-box approach to calculate the eigenvalues of each of the hessians:

eigenvals1 = tf.self_adjoint_eigvals(hessians[0])
eigenvals2 = tf.self_adjoint_eigvals(hessians[1])
eigenvals3 = tf.self_adjoint_eigvals(hessians[2])
eigenvals4 = tf.self_adjoint_eigvals(hessians[3])

1,2, and 4 work, but the 3rd one bombs out, complaining that Dimensions must be equal, but are 2 and 1 for 'SelfAdjointEigV2_2' (op: 'SelfAdjointEigV2') with input shapes: [2,1,2,1]. Should I just reshape the hessian somehow and carry on, or am I on the wrong track entirely?

anna-earwen
  • 189
  • 13

1 Answers1

3

After some fiddling, I have figured out that, given n*m matrix of input variables, TensorFlow's tf.hessians produces [n,m,n,m] tensor, which can be reshaped into square [n*m, n*m] Hessian matrix as follows:

sq_hess = tf.reshape(hessians[0], [tf.size(weights['h1']), tf.size(weights['h1'])])

Further, one can calculate the eigenvalues of the resulting square hessian:

eigenvals = tf.self_adjoint_eigvals(sq_hess)

This might be trivial, but it took me some time to wrap my head around this. I believe the behaviour of tf.hessians is not very well documented. Once you put together the dimensionalities, though, everything makes sense!

anna-earwen
  • 189
  • 13
  • `tf.hessians()` will provide two matrices which correspond to upper and lower block diagonal matrices. How do you compute the the full Hessian’s two off-diagonal block matrices? look at Page 6 in [heref](https://arxiv.org/pdf/1905.05559v1.pdf). – ARAT May 16 '19 at 18:24