2

I am preparing to do topic modeling via Mallet and have finished pulling the raw datasets. Before I import and start modeling, I need to take some steps to clean and streamline the texts, of course. I have my lists of stopwwords ready and I know that I can remove punctuation, digits etc. easily with Excel. What I am a little fuzzy about is stemming and lemmatizing. Not on the concept itself but rather what the best approach would be.

To give a better overview, here is what I would like to do:

  • standardize inconsistencies in spelling, e.g. topicmodeling -> topic modeling
  • remove extra whitespaces from words, e.g. two whitespaces in a row
  • stem and lemmatize

Based on experience, can anyone recommend the best approach to these three, especially to the last though? Is there an app that I can use for that?

Many thanks in advance!

Glorifier
  • 31
  • 1

1 Answers1

2

See what happens without these interventions first.

Whitespace and punctuation usually isn't a problem, but you might want to make sure that text contains no tabs or newlines, as these can confuse the data import functions. There is a common problem when importing into something like excel that pays attention to quotation marks, where it can interpret many lines as a single document if quotes aren't matched.

The problem with stemming, lemmatization, and spelling regularization is that they have the same objective as the topic model itself. Its goal is to combine semantically similar words based on context, so it actually doesn't have a problem with the kind of variation you see in English. For other languages with lots of morphology you might need something more advanced. But in most cases you are really just making the model's job harder.

One way to use a stemmer is to stem after modeling. People often think they need stemmers because they see multiple small variants of a word in the model output. I would argue that this is a sign the model is working, but I can see that it might not be the best interface. In this case you might notice that certain words map to the same stem, and only show the original form of the most frequent one.

In my experience the most effective interventions you can make are cleaning up problems in input (like hyphen- ated words) and converting significant multi-word terms to single terms (like topic modeling to topic_modeling).

David Mimno
  • 1,836
  • 7
  • 7
  • Hi David! thank you for your answer. To give some context, my text data are Youtube comments. Typically, you'll find quite a lot emoticons/emojis in them so cleaning up really does help, even if its more for my personal overview than for the model itself. My text language is German so lemmatizing would indeed help with the many variations we have. I have been reading about lemmatizing with spaCy and NLTK. As far as I understand it, I would need to install Gensim Wrapper for Mallet first to load either óf them into Mallet. Is this correct or do they work in independent environments? – Glorifier Jun 29 '21 at 20:12
  • (was running out of characters in the first comment) Also, you are recommending cleaning up hyphen-ated words and converting multi-word terms. The latter is especially interesting to me. Is there a way to do this in Mallet or is there an extra package you can recommend? Many thanks! – Glorifier Jun 29 '21 at 20:19