0

I am using tf.matrix_inverse() to compute the inverse of matrix. Sometimes, the matrix is very close to singular but not singular. I wonder in Tensorflow what's the threshold (or how close determinant of matrix to 0) for tf.matrix_inverse() to define it's a "singular matrix"?

Ruofan Kong
  • 1,060
  • 1
  • 17
  • 34
  • 1
    If any eigenvalue divided by spectral radius is below 1e-7 then this eigenvalue is within machine precision of 0 (in float32) and inverting matrix numerically doesn't make sense. Surprisingly wide-spread magic number -- add identity*0.001 to your matrix to make inversion work – Yaroslav Bulatov Aug 03 '17 at 23:13
  • @YaroslavBulatov Thanks for your answer!! Just have 2 more questions :) 1. According to doc of `tf.matrix_inverse()`, looks it's using LU decomposition to compute inverse (not svd). Why is it related to eigen-values? 2. if 1e-7 is threshold by spectral radius, is it supposed to just add identity * 1e-7? – Ruofan Kong Aug 03 '17 at 23:21
  • 1
    1) because eigenvalues that determine invertibility of a matrix. If eigenvalue is too close to zero, all inversion methods will have a problem 2) you could add 1e-7 to make it invertible although result will jump around a lot – Yaroslav Bulatov Aug 03 '17 at 23:48
  • @YaroslavBulatov I'm going to solve linear regression problem `Ax = y` in which row(A) > column(A), and I am using `inv(A^T * A + lambda * I) * A^T * y` with `lambda = 1e-7` to solve. Instead of getting singular error message from `tf.matrix_inverse()` (to compute `inv(A^T * A + lambda * I)`), I am getting wrong answer. If I increased lambda=1e-3, the answer is correct, but accuracy is very low. So I wonder if there is way to get better accuracy without singularity (if I don't change tf.float32 to tf.float64). – Ruofan Kong Aug 04 '17 at 01:38
  • 1
    matrix inverse is not ideal choice for solving linear regression, why not use linsolve directly? – Yaroslav Bulatov Aug 04 '17 at 05:31
  • @YaroslavBulatov 1. My algorithm does not allow me to use gradient based approach to solve. 2. I also tried 'tf.matrix_solve_ls()' with 'fast=True', but will get singular error using float32 with small 'l2_regularizer', if 'l2_regularizer' is bit larger (like 1e-3), still accuracy is too low... If I use 'fast=False', it would run very slow. So I wonder if there is any other way to do it (Not changing from float32 to float 64) – Ruofan Kong Aug 04 '17 at 08:08
  • 1
    What about using pseudo inverse? (Using svd) – Yaroslav Bulatov Aug 04 '17 at 16:28
  • @YaroslavBulatov hmm... That should work. Is svd supported on GPU? – Ruofan Kong Aug 09 '17 at 01:08
  • 1
    Not yet, but you can patch in this PR and rebuild -- https://github.com/tensorflow/tensorflow/pull/11878 – Yaroslav Bulatov Aug 09 '17 at 01:14
  • Thanks!! BTW, do you think it's going to be included in the next TF release? – Ruofan Kong Aug 09 '17 at 01:19
  • 1
    yes, you can see in the PR comments that it's close to being merged into master, which would make it available in a nightly binary. Releases happen every 6 weeks, so it'll be a few weeks for official release inclusion – Yaroslav Bulatov Aug 09 '17 at 01:22

0 Answers0