3

I have a list of size 208 (208 arrays of sentences), that looks like:

all_words = [["this is a sentence ... "] , [" another one hello bob this is alice ... "] , ["..."] ...]

I want to get the words with the highest tf-idf values. I created a tf-idf matrix:

from sklearn.feature_extraction.text import TfidfVectorizer

tokenize = lambda doc: doc.split(" ")
sklearn_tfidf = TfidfVectorizer(norm='l2', tokenizer=tokenize, ngram_range=(1,2))
tfidf_matrix = sklearn_tfidf.fit_transform(all_words)
sentences = sklearn_tfidf.get_feature_names()

dense_tfidf = tfidf_matrix.todense()

Now I don't know how to get the words with the highest tf-idf values.

Each column of the dense_tfidf represents a word/2-words. (the matrix is 208x5481)

When I summed each column, it didn't really help - got the same result of a simple top words (I guess because it's the same as a simple word count).

How can I get the words with the highest tf-idf value? Or how can I normalize it wisely?

sheldonzy
  • 5,505
  • 9
  • 48
  • 86

1 Answers1

3

Had a similar issue but found this at https://towardsdatascience.com/multi-class-text-classification-with-scikit-learn-12f1e60e0a9f, just change the X and y inputs based on your dataframe. The code from the blog is below. Sklearn's doc helped me: http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html

from sklearn.feature_selection import chi2
import numpy as np
N = 2
for Product, category_id in sorted(category_to_id.items()):
features_chi2 = chi2(features, labels == category_id)
indices = np.argsort(features_chi2[0])
feature_names = np.array(tfidf.get_feature_names())[indices]
unigrams = [v for v in feature_names if len(v.split(' ')) == 1]
bigrams = [v for v in feature_names if len(v.split(' ')) == 2]
print("# '{}':".format(Product))
print("  . Most correlated unigrams:\n. {}".format('\n. '.join(unigrams[-N:])))
print("  . Most correlated bigrams:\n. {}".format('\n. '.join(bigrams[-N:])))
Ben M
  • 51
  • 8