0

My understanding of the work flow is to run LDA -> Extract keywards (e.g. the top few words for each topics), and hence reduce dimension -> some subsequent analysis.

My question is, if my overall purpose is to give topic to articles in an unsupervised way, or clustering similar documents together, then a running of LDA will take you directly to the goal. Why do you reduce the dimension and then pass it to subsequent analysis? If you do, what sort of subsequent analysis can you do after LDA?

Also, a bit unrelated question -- is it better to ask this question here or at cross validated?

nobody
  • 815
  • 1
  • 9
  • 24

2 Answers2

0

I think cross validated is a better place for these kinds of questions. Anyhow, there are simple explanations about why we need dimension reduction:

  1. Without dimension reduction, vector operations are not computable. Imagine a dot product between two vector with dimension in size of your dictionary! really?
  2. Each number carry more dense amount of information after reducing the dimension. Which it usually leads to less noise. Intuitively, you only kept useful information.
Mehdi
  • 4,202
  • 5
  • 20
  • 36
0

You should rethink your approach, since you are mixing probabilistic methods (LDA) with Linear Algebra (dimensional reduction). When you feel more comfortable with Linear Algebra, consider Non Negative Matrix Factorisation.

Also note that your topics already constitute the reduced dimensions, there is no need to jump back to the extracted top words in the topics.

Sir Cornflakes
  • 675
  • 13
  • 26