I have a corpus with around 1,500,000 documents of titles and abstracts from scientific research projects within STEM. I used Mallet https://mimno.github.io/Mallet/transforms to fit models from 10 to 790 topics in 10 topics increments (I allow for hyperparameter optimization). I ran three replicates with differing seeds using 70% of the corpus for training and 30% for validation. The corpus was pre-processed to remove stop-words, lemmatise, etc.
I want to choose the model with best number of topics. I understand that coherence and perplexity are two quantitative measures of model performance that should allow me to do this. They measure two 'separate' types of performance. Coherence measures the ability of the model finding topics that are the closest to human interpretability. While perplexity measures the ability of the model to classify unseen data with a trained model.
However, when I calculate these two methods they show opposing results with no clear optimum in either.
I was hoping that both would show some aggrement in trend, so i could choose what number of topics to work with (see figure below).
