I am using gensim's LDA and trying to see the perplexity for a certain number of topics.
Perplexity for 1 : -7.903370624873305
Coherence Score for 1 : 0.8044880331838007
Perplexity for 2 : -8.269065851934347
Coherence Score for 2 : 0.7999767615350039
Perplexity for 3 : -8.55527218635052
Coherence Score for 3 : 0.7957008853871509
Perplexity for 4 : -8.769445203605587
Coherence Score for 4 : 0.7950156446303915
Perplexity for 5 : -8.953285444837803
Coherence Score for 5 : 0.7895563460199423
Perplexity for 6 : -9.118557691084591
Coherence Score for 6 : 0.7886679109232323
Perplexity for 7 : -9.347579422466403
Coherence Score for 7 : 0.7862291131217295
Perplexity for 8 : -10.515445720592052
Coherence Score for 8 : 0.7870887700340385
Perplexity for 9 : -15.37602605337135
Coherence Score for 9 : 0.793312181374788
How do I judge here? If it is supposed to be closer to 0, then it would mean that with 1 topic its the best but that doesn't really make sense since I had over 100000 words for training the LDA. Should I choose somewhere in between?
