6

The following code for the CIFAR dataset after GCN:

xtx = np.dot(dataset.train_data[i].transpose(), dataset.train_data[i])
e, q = np.linalg.eigh(xtx)
print(np.max(e), np.min(e))

Produces the following output:

2.65138e+07 -0.00247511

This is inconsistent given that xtx is symmetric positive-semi definite. My guess is that this might be due to applying GCN earlier, but still the minimum eigenvalue is not even that close to 0?

Update: So the condition number of my matrix is 8.89952e+09. I've actually have forgotten before to take out the mean so now the maximum eigenvalue is ~573, while the minimum is -7.14630133e-08. My question is that I'm trying to do ZCA. In this case how should I proceed? Add a diagonal petrubtion to xtx or to the Eigenvalues?

steven
  • 226
  • 1
  • 13
Alex Botev
  • 1,369
  • 2
  • 19
  • 34
  • What is the data type of `xtx` (check `xtx.dtype`)? Is it single precision (`np.float32`)? – Warren Weckesser Feb 24 '16 at 06:12
  • i would say it's pretty close to zero when you consider there are 10 orders of magnitude between the max and min eigenvalue, as @WarrenWeckesser notes this is well within the margin of error for 32 bit float (7-9 digits of precision on average) – maxymoo Feb 24 '16 at 06:15
  • Please post also the result of `numpy.linalg.cond(xtx)` – gg349 Feb 24 '16 at 06:16
  • @maxymoo, you are assuming that all eigenvalues have the same precision, which of course would depend on how `eigh` works. I doubt this, and tried to find this out quick in the lapack code, but with no success. – gg349 Feb 24 '16 at 06:31
  • I've updated the questions – Alex Botev Feb 24 '16 at 21:57

1 Answers1

0

If you eigenvalues accross several orders of magnitude you can use the svd decomposition so that X.T @ X = (U @ S @ V.T).T @ (U @ S @ V.T) = V @ S @ S @ V.T.

_, s, v = np.linalg.svd( dataset.train_data[i] )
e = s[:len(v)]**2

Now e is guaranteed to be positive, and, suppose that you calculated something around 5e3 for the higher singular value and that rounding errors limits the accuracy of the other SVD at 1e-7, this means that when you calculate e[1] = s[1]**2 you will actually something at the order of 1e-14.

Bob
  • 13,867
  • 1
  • 5
  • 27