0

I don't understand SVD at algorithm level. But I do know that people use it to reduce dimension. I have two co-occurrence matrix (dimension is 50,000 words by 50,000 words) that store information on the number of times any words were used together. The two matrix was constructed in the same way (the row and column matches with each other).

If I perform SVD to reduce their dimension to 300, will the new columns of two matrixs still match with each other and represent the same linguistic structure?

Ying Li
  • 1
  • 2

1 Answers1

0

The singular value decomposition is a generalized eigendecomposition. It takes any matrix that is m by n and finds an orthogonal basis for it. The reason why it will reduce the dimension is there is a special type of SVD called the truncated SVD. In linear algebra, you learn there is something called the rank of the matrix.

The rank represents the maximum number of linearly independent vectors in a matrix. The SVD employs something called Gram-Schmidt to get this orthogonal basis which are (a type of) linearly independent vectors.

The co-occurrence matrix you are talking about is used NLP (natural language processing). Here it indicates whether two words occur together in a specific context together. You pretty obviously can't reduce the dimension beyond a certain point. The point of the SVD is to fundamentally represent the occurences of words in a specific way.

will it still match their structure - yes it will. However, say for instance if the rank of the matrix is 300 and you go to 250. Then you will lose information. There is a theorem on this about low rank matrix approximations. It is called the eckart-young-mirsky theorem.

If you go beneath this point you will actually have bounds on the error of your approximation in linguistic structure.