data frame of tfidf with Python

Question

I have to classify some sentiments my data frame is like this

Phrase                      Sentiment    
is it  good movie          positive    
wooow is it very goode      positive    
bad movie                  negative

I did some preprocessing as tokenisation stop words stemming etc ... and I get

Phrase                      Sentiment    
[ good , movie  ]        positive    
[wooow ,is , it ,very, good  ]   positive 
[bad , movie ]            negative

I need finally to get a dataframe in which the line are the text which the value is the tf_idf and the columns are the words like that

good     movie   wooow    very      bad                Sentiment
tf idf    tfidf_  tfidf    tf_idf    tf_idf               positive
(same thing for the 2 remaining lines)

score 7 · Accepted Answer · edited May 23 '17 at 12:17

7

I'd use sklearn.feature_extraction.text.TfidfVectorizer, which is specifically designed for such tasks:

Demo:

In [63]: df
Out[63]:
                   Phrase Sentiment
0       is it  good movie  positive
1  wooow is it very goode  positive
2               bad movie  negative

Solution:

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

vect = TfidfVectorizer(sublinear_tf=True, max_df=0.5, analyzer='word', stop_words='english')

X = vect.fit_transform(df.pop('Phrase')).toarray()

r = df[['Sentiment']].copy()

del df

df = pd.DataFrame(X, columns=vect.get_feature_names())

del X
del vect

r.join(df)

Result:

In [31]: r.join(df)
Out[31]:
  Sentiment  bad  good     goode     wooow
0  positive  0.0   1.0  0.000000  0.000000
1  positive  0.0   0.0  0.707107  0.707107
2  negative  1.0   0.0  0.000000  0.000000

UPDATE: memory saving solution:

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

vect = TfidfVectorizer(sublinear_tf=True, max_df=0.5, analyzer='word', stop_words='english')

X = vect.fit_transform(df.pop('Phrase')).toarray()

for i, col in enumerate(vect.get_feature_names()):
    df[col] = X[:, i]

UPDATE2: related question where the memory issue was finally solved

edited May 23 '17 at 12:17

Community

1
1

answered Jan 28 '17 at 09:19

MaxU - stand with Ukraine

205,989
36
386
419

i have a problem memory error in this line r = pd.DataFrame(X.toarray(), columns=vect.get_feature_names()) – Amal Kostali Targhi Jan 28 '17 at 23:16
@AmalKostaliTarghi, i've updated my answer - please check whether it helped – MaxU - stand with Ukraine Jan 28 '17 at 23:25
i think the problem is in df = pd.DataFrame(X.toarray(), columns=vect.get_feature_names()) because my corpus contain more than 10000 line and after applying tfidf THE MATRix became 156060x11780 so i think there a problem there – Amal Kostali Targhi Jan 29 '17 at 00:20
@AmalKostaliTarghi, yes, `156060*11780*8/1024**3` is approx. 14GiB and it's copied for a very short time when `DataFrame()` constructor is called – MaxU - stand with Ukraine Jan 29 '17 at 00:27
@AmalKostaliTarghi, i can imagine one ugly solution - adding columns in loop. It'll be slower, but should be memory-saving – MaxU - stand with Ukraine Jan 29 '17 at 00:35
Thank you but how can i add columns by loop – Amal Kostali Targhi Jan 29 '17 at 00:37
@AmalKostaliTarghi, please check UPDATE section – MaxU - stand with Ukraine Jan 29 '17 at 00:38
thank you but the problem is here: X = vect.fit_transform(df.pop('Phrase')).toarray() – Amal Kostali Targhi Jan 29 '17 at 00:40
@AmalKostaliTarghi, i think it's a good idea to try to use sparse matrix/DataFrame, but i don't know yet how to do that... – MaxU - stand with Ukraine Jan 29 '17 at 00:51
@AmalKostaliTarghi. try this: `pd.SparseDataFrame(X.todense(), columns=vect.get_feature_names(), index=df.index, , default_fill_value=0)` where `X = vect.fit_transform(df.pop('Phrase'))` – MaxU - stand with Ukraine Jan 29 '17 at 00:54

score 6 · Answer 2 · edited Apr 10 '22 at 08:04

setup

df = pd.DataFrame([
        [['good', 'movie'], 'positive'],
        [['wooow', 'is', 'it', 'very', 'good'], 'positive'],
        [['bad', 'movie'], 'negative']
    ], columns=['Phrase', 'Sentiment'])

df

                        Phrase Sentiment
0                [good, movie]  positive
1  [wooow, is, it, very, good]  positive
2                 [bad, movie]  negative

Calculating term frequency tf

# use `value_counts` to get counts of items in list
tf = df.Phrase.apply(pd.value_counts).fillna(0)
print(tf)

   bad  good   is   it  movie  very  wooow
0  0.0   1.0  0.0  0.0    1.0   0.0    0.0
1  0.0   1.0  1.0  1.0    0.0   1.0    1.0
2  1.0   0.0  0.0  0.0    1.0   0.0    0.0

Calculating inverse document frequency idf

# add one to numerator and denominator just incase a term isn't in any document
# maximum value is log(N) and minimum value is zero
idf = np.log((len(df) + 1 ) / (tf.gt(0).sum() + 1))
idf

bad      0.693147
good     0.287682
is       0.693147
it       0.693147
movie    0.287682
very     0.693147
wooow    0.693147
dtype: float64

tfidf

tdf * idf

        bad      good        is        it     movie      very     wooow
0  0.000000  0.287682  0.000000  0.000000  0.287682  0.000000  0.000000
1  0.000000  0.287682  0.693147  0.693147  0.000000  0.693147  0.693147
2  0.693147  0.000000  0.000000  0.000000  0.287682  0.000000  0.000000

data frame of tfidf with Python

2 Answers2

Linked