22

Disclaimer
This is not strictly a programming question, but most programmers soon or later have to deal with math (especially algebra), so I think that the answer could turn out to be useful to someone else in the future.

Now the problem
I'm trying to check if m vectors of dimension n are linearly independent. If m == n you can just build a matrix using the vectors and check if the determinant is != 0. But what if m < n?

Any hints?

See also this video lecture.

Bill the Lizard
  • 398,270
  • 210
  • 566
  • 880
tunnuz
  • 23,338
  • 31
  • 90
  • 128

8 Answers8

22

Construct a matrix of the vectors (one row per vector), and perform a Gaussian elimination on this matrix. If any of the matrix rows cancels out, they are not linearly independent.

The trivial case is when m > n, in this case, they cannot be linearly independent.

David Hanak
  • 10,754
  • 3
  • 31
  • 39
  • 2
    Could you please explain better your solution? I should execute a gaussian elimination on what exactly? – tunnuz Feb 10 '09 at 12:10
  • 1
    Let's say you have the 2 vectors (2 3) (4 6). They map to the following set of equations: `2x + 3y = a` and `4x + 6y = b`. If you try a gaussian elimination of x, you end up with `0x + 0y = 2a - b`. Having the zeros indicates that the two vectors are not independant. Generalize for `M` and `N`. – Pierre Feb 10 '09 at 12:16
  • So this has at least `O(n^3)` complexity (depending on how we measure). Do you know if there is a (perhaps randomized) `O(n^2)` algorithm? – Thomas Ahle Feb 25 '14 at 15:16
7

Construct a matrix M whose rows are the vectors and determine the rank of M. If the rank of M is less than m (the number of vectors) then there is a linear dependence. In the algorithm to determine the rank of M you can stop the procedure as soon as you obtain one row of zeros, but running the algorithm to completion has the added bonanza of providing the dimension of the spanning set of the vectors. Oh, and the algorithm to determine the rank of M is merely Gaussian elimination.

Take care for numerical instability. See the warning at the beginning of chapter two in Numerical Recipes.

jason
  • 236,483
  • 35
  • 423
  • 525
3

If m<n, you will have to do some operation on them (there are multiple possibilities: Gaussian elimination, orthogonalization, etc., almost any transformation which can be used for solving equations will do) and check the result (eg. Gaussian elimination => zero row or column, orthogonalization => zero vector, SVD => zero singular number)

However, note that this question is a bad question for a programmer to ask, and this problem is a bad problem for a program to solve. That's because every linearly dependent set of n<m vectors has a different set of linearly independent vectors nearby (eg. the problem is numerically unstable)

jpalecek
  • 47,058
  • 7
  • 102
  • 144
2

I have been working on this problem these days.

Previously, I have found some algorithms regarding Gaussian or Gaussian-Jordan elimination, but most of those algorithms only apply to square matrix, not general matrix.

To apply for general matrix, one of the best answers might be this: http://rosettacode.org/wiki/Reduced_row_echelon_form#MATLAB

You can find both pseudo-code and source code in various languages. As for me, I transformed the Python source code to C++, causes the C++ code provided in the above link is somehow complex and inappropriate to implement in my simulation.

Hope this will help you, and good luck ^^

Edward
  • 71
  • 5
1

Sorry man, my mistake...

The source code provided in the above link turns out to be incorrect, at least the python code I have tested and the C++ code I have transformed does not generates the right answer all the time. (while for the exmample in the above link, the result is correct :) -- )

To test the python code, simply replace the mtx with

[30,10,20,0],[60,20,40,0]

and the returned result would be like:

[1,0,0,0],[0,1,2,0]

Nevertheless, I have got a way out of this. It's just this time I transformed the matalb source code of rref function to C++. You can run matlab and use the type rref command to get the source code of rref.

Just notice that if you are working with some really large value or really small value, make sure use the long double datatype in c++. Otherwise, the result will be truncated and inconsistent with the matlab result.

I have been conducting large simulations in ns2, and all the observed results are sound. hope this will help you and any other who have encontered the problem...

Edward
  • 71
  • 5
1

If computing power is not a problem, probably the best way is to find singular values of the matrix. Basically you need to find eigenvalues of M'*M and look at the ratio of the largest to the smallest. If the ratio is not very big, the vectors are independent.

mattiast
  • 1,934
  • 12
  • 18
1

Another way to check that m row vectors are linearly independent, when put in a matrix M of size mxn, is to compute 

det(M * M^T)

i.e. the determinant of a mxm square matrix. It will be zero if and only if M has some dependent rows. However Gaussian elimination should be in general faster.

Fanfan
  • 1,116
  • 5
  • 7
0

A very simple way, that is not the most computationally efficient, is to simply remove random rows until m=n and then apply the determinant trick.

  • m < n: remove rows (make the vectors shorter) until the matrix is square, and then
  • m = n: check if the determinant is 0 (as you said)
  • m < n (the number of vectors is greater than their length): they are linearly dependent (always).

The reason, in short, is that any solution to the system of m x n equations is also a solution to the n x n system of equations (you're trying to solve Av=0). For a better explanation, see Wikipedia, which explains it better than I can.

Mark
  • 18,730
  • 7
  • 107
  • 130