1

I am reading the book "Introduction to linear algebra" by Gilbert Strang. The section is called "Orthonormal Bases and Gram-Schmidt". The author several times emphasised the fact that with orthonormal basis it's very easy and fast to calculate Least Squares solution, since Qᵀ*Q = I, where Q is a design matrix with orthonormal basis. So your equation becomes x̂ = Qᵀb.

And I got the impression that it's a good idea to every time calculate QR decomposition before applying Least Squares. But later I figured out time complexity for QR decomposition and it turned out to be that calculating QR decomposition and after that applying Least Squares is more expensive than regular x̂ = inv(AᵀA)Aᵀb.

  1. Is that right that there is no point in using QR decomposition to speed up Least Squares? Or maybe I got something wrong?

  2. So the only purpose of QR decomposition regarding Least Squares is numerical stability?

mathgeek
  • 303
  • 1
  • 9

1 Answers1

1

There are many ways to do least squares; typically these vary in applicability, accuracy and speed.

Perhaps the Rolls-Royce method is to use SVD. This can be used to solve under-determined (fewer obs than states) and singular systems (where A'*A is not invertible) and is very accurate. It is also the slowest.

QR can only be used to solve non-singular systems (that is we must have A'*A invertible, ie A must be of full rank), and though perhaps not as accurate as SVD is also a good deal faster.

The normal equations ie

compute P = A'*A
solve P*x = A'*b

is the fastest (perhaps by a large margin if P can be computed efficiently, for example if A is sparse) but is also the least accurate. This too can only be used to solve non singular systems.

Inaccuracy should not be taken lightly nor dismissed as some academic fanciness. If you happen to know that the problems ypu will be solving are nicely behaved, then it might well be fine to use an inaccurate method. But otherwise the inaccurate routine might well fail (ie say there is no solution when there is, or worse come up with a totally bogus answer).

I'm a but confused that you seem to be suggesting forming and solving the normal equations after performing the QR decomposition. The usual way to use QR in least squares is, if A is nObs x nStates:

    decompose A as A = Q*(R ) 
                         (0 )
    transform b into b~ = Q'*b
    (here R is upper triangular)
    solve R * x =  b# for x, 
    (here b# is the first nStates entries of b~)
dmuir
  • 4,211
  • 2
  • 14
  • 12
  • Thank you for the answer! But you state that I can't find least squares solution for [this system](https://i.stack.imgur.com/pnuJB.png), using QR decomposition? – mathgeek Nov 06 '21 at 16:19
  • I've updated my answer. You can use QR% for the system you described. It is not whether A is singular or not, but whether A'*A is singular or not that matters – dmuir Nov 06 '21 at 17:08
  • Yes, that's true. When A has independent columns A'*A is always invertible. Now that we agree about ability to use QR decomposition before least squares could you directly address my two questions, please? – mathgeek Nov 06 '21 at 17:12
  • Also, please update your comment "ie A must be of full rank". A doesn't have to be of full rank. It only needs to be of full column rank. – mathgeek Nov 06 '21 at 17:15
  • @mathgeek the maximum rank that an mxn matrix can have is min(m,n). For A'A to be non singular we must have (where A is mxn) m>=n and A of rank n – dmuir Nov 06 '21 at 20:09
  • I tried to answer your question about doing QR before by pointing out that it was confused. The drill is not that you do a qr and then form the normal equations and solve them, but rather, as in my final oaragraph, do the QR and then solve for the states without forming the normal equations. This is likely to be somewhat slower, but also more accurate, than solving the the problem via the normal equations. – dmuir Nov 06 '21 at 20:14
  • You're right about full rank, sorry. So you state that QR decomposition is used only for numerical accuracy and not for speeding up calculations? – mathgeek Nov 06 '21 at 20:23
  • Yes, but of course accuracy is important or rather inaccuracy can be disastrous, so my attitude is to default to QR (or SVD) and only consider using the normal equations, as an optimisation, if I am confident that the problem is nice enough that the inaccuracy won't be bad news. – dmuir Nov 06 '21 at 20:45
  • Thank you very much! As a bonus question, could you clarify something? As I understand, all the decompositions (LU, QR, SVD) slow down solving Ax = b. Are they actually used only for numerical stability? I'm a bit perplexed at whether they only used for better accuracy and not for speeding up calculations. – mathgeek Nov 06 '21 at 21:19
  • @mathgeek How would you solve Ax = b without using one of the factorisations? The standard methods to solve would use one of those factorisations. – dmuir Nov 11 '21 at 12:41
  • I can always find inverse. But as I know that's not stable. That's why I'm asking. Are all these factorisations used only for numerical stability? – mathgeek Nov 11 '21 at 18:43
  • @mathgeek How do you find the inverse? The usual way, and the way most inversion routines work under the hood, is to factorise and then invert the factors and multiply those inverses, Inversion has a bad dame compared with solving as being slower and less accurate. – dmuir Nov 13 '21 at 08:50