Counting Words in a Dataframe in Python

Question

I have imported a CSV file to Python using pandas. The file consists of 3 columns and 498 rows. I only need to have a word count for 1 column named "Description". I have cleaned the file by transforming the column "Description" to lower-case, removing english stopwords and splits.

IN:

    import pandas as pd

    df = pd.read_csv("capex_motscles.csv")

    from nltk.corpus import stopwords
    stop = stopwords.words('english') 

    Description3 = df['Description'].str.lower().apply(lambda x: 
    ''.join([word for word in str(x).split() if word not in (stop)]))

    print(Description3)

OUT:

    0      crazy mind california medical service data base...
    1      california licensed producer recreational & medic...
    2      silicon valley data clients live beyond status...
    3      mycrazynotes inc. announces $144.6 million expans...
    4      leading provider sustainable energy company prod ...
    5      livefreecompany founded 2005, listed new york stock...

I've provided 5 rows from "print(Description3)". I have 498 rows in total, and as mentioned, I need to count the word frequencies. Any help would be greatly appreciated, thank you for your time!

Thank you, I will take a look into Counter. I did not try nltk as I am still not proficient in Python. But I will look into it as well. — OnThaRise, May 14 '19 at 17:33

Simon Rogers · Answer 1 · 2019-05-14T16:50:36.927

1

Do you mean something like this?

df['Description3'] = df['Description'].str.lower().apply(lambda x: 
                             ''.join([word for word in str(x).split() if word not in (stop)]))

df['Description3'].str.split(expand=True).stack().value_counts()

edited May 14 '19 at 16:50

answered May 14 '19 at 16:39

Simon Rogers

344
1
10

Thank you. It did count the words as I would have wanted. I will try and clean my column a little bit more to get rid of inconsistencies. – OnThaRise May 14 '19 at 17:34

Counting Words in a Dataframe in Python

1 Answers1