I am using seven lexicons to calculate sentimental scores on a data set containing forum posts. Apart from removing all noise such as whitespace, special char, digits and stopwords, why is it also important to stem the words?
I am using Harvard.IV, Qdap, Henry's Financial dictionary and Loughran-McDonald Financial dictionary from SentimentAnalysis package, as well as AFINN, NRC and BING dictionaries.
It is an arguable statement that stemming is important for sentiment analysis.
First of all, different terms with different sentiment values or senses are formed into the same stem. You can check Porter Stemmer on Harvard General Inquirer. An example from page is closeness and close is stemmed into the same term, where one term has a positive sense at the first place and the other has negative.
Another point that one can argue about stemming is important for sentiment analysis is that syntactic and semantic processing is almost impossible after stemming. For example, if one would like to pos-tag terms, create parse trees for the sentences or find triggers and scopes of negation; stemming is not the process to be considered applying.
For example, if we have the term pretty in our sentence, and we want to determine if the term is an adjective or an adverb, where pretty woman is positive and pretty awful is negative, applying stemming breaks pos-tagging.
Cheers
Because this allows you to reduce noise in your data. The process of stemming reduce inflectional forms and related forms to the common base of a word. Please check this very informative tutorial from The Stanford Natural Language Processing Group
