In the example 18.4 of the book Introduction to Information Retrieval. The term-document matrix is decomposed using SVD. My question is why Σ is a 5*5 matrix in the example? Shouldn't it be a 5*6 matrix? Is it wrong?
Here is the link of the Chapter 18 of the book Introduction to Information Retrieval. Thanks!
The book is correct. A term document matrix (of dimension DxT) is split into a product of three matrices. The middle matrix (denoted as \Sigma in the book) is the key matrix whose dimension is TxT (T=5 in the example).
Intuitively, you can think of this matrix as denoting relationship between terms. In the best case, all the column vectors of this matrix should be linearly independent meaning that this forms the basis vector in the term space and there is no dependence between the terms. However, this is not true in practice. You'll find that the rank of this matrix is typically a few orders of magnitude less than T (say T'), meaning that there are T-T' linearly dependent column vectors in this matrix.
One can then take a lower order approximation of this matrix by considering only a T'xT' term matrix. In effect, you take the principal eigen values of the matrix and project your vectors on these eigen vectors (treated as new basis) using rotation and scaling. That's precisely what spectral decomposition or PCA (or LSA) does.
