1

My goal is to identify topics of tweets and visualize how the distribution of topics changed over time. As far as I know, the best way to do it is with the stm package but I have some problems with it. So, my only option is to do a simple LDA.

Based on the topic shares for each of the tweets, I aggregated the shares of topics per year and compared each topic share versus the total of each year (the same way it is done here https://towardsdatascience.com/thats-mental-using-lda-topic-modeling-to-investigate-the-discourse-on-mental-health-over-time-11da252259c3). The final visualization looks similar to this: topics over time

My question is f it is possible to visualize topics over time with LDA what is the point of doing it in STM? Are there any important differences?

glts
  • 21,808
  • 12
  • 73
  • 94
Olyalya
  • 11
  • 1

1 Answers1

1

Post hoc analysis is a fine way to measure topic prevalence over time. LDA doesn't explicitly learn parameters that represent the relationship between years and topics, but as you've found, that doesn't mean that there isn't a relationship.

LDA often doesn't work well for short documents like tweets. You might also try k-means.

STM is helpful if you want to make an argument about the relationship in the way that you would with any other regression model. The advantage would be that you might get topics that are more aligned with your covariates, but it's usually not necessary.

Visually, I really don't like streamplots for topic over time results. For example, because of the spike in Art in 2000, you can't tell whether any of the other topics is changing in that year because they're all "shoved aside" from above. Giving each topic its own area chart makes it much easier to see individual trends.

David Mimno
  • 1,836
  • 7
  • 7
  • Thank you a lot! Maybe, you know how to make these charts? I have some graphs that I made with the dfrtopics package the same as it was done here (https://stackoverflow.com/questions/48764778/topics-proportion-over-time-using-mallet-lda). However, with this function (topic_series) the majority of topics have a low proportion at the beginning of the period. I assume it is because there were few tweets that time. So, these graphs are quite confusing. – Olyalya May 11 '21 at 07:35