Counting the words uniquely in a dataframe

Question

for filename in glob.glob(os.path.join(folder_path, '*.html')):
  with open(filename, 'r',encoding='utf-8') as f:
    text = f.read()
    print (filename)
    #print (text)
    patent = BeautifulSoup(text)
    cleantext = patent.get_text()
    clean_lower=cleantext.lower()
    for char in clean_lower:
    >> if char not in punctuations:
       no_punct = no_punct + char
    for word in dictionary:
      >>if word in no_punct:
        >>>wordlist.append(word)
        >>>countlist.append(no_punct.count(word))

print(wordlist,countlist)
df = pd.DataFrame({'word':wordlist, 'count':countlist})
df.columns=['word','count']
df=df.set_index('word')
print(df)

['steam', 'heating', 'horizontal well', 'electromagnetic', 'single well', 'steam', 'foam', 'heating', 'horizontal well', 'solvent', 'hexane', 'electromagnetic', 'steam foam', 'surfactant', 'single well', 'miscible'] [84, 9, 4, 2, 1, 89, 2, 10, 4, 5, 7, 2, 1, 106, 1, 1]
                 count
word                  
steam               84
heating              9
horizontal well      4
electromagnetic      2
single well          1
steam               89
foam                 2
heating             10
horizontal well      4
solvent              5
hexane               7
electromagnetic      2
steam foam           1
surfactant         106
single well          1
miscible             1

I am not getting a unique output, could someone tell me where am I making a mistake in the loop? The count of word steam should be 89 but i want it printed only once.

How to plot a bar graph showing the phrase with highest count rather than in random order? — astronaut, Mar 23 '20 at 17:42

score 0 · Answer 1 · answered Mar 23 '20 at 17:28

0

Use df.drop_duplicates() to delete duplicate rows

answered Mar 23 '20 at 17:28

Suraj

2,253
3
17
48

I am not able to try the Beautifulsoup code, but hope you find this link useful https://stackoverflow.com/a/24396780/10077354 – Suraj Mar 23 '20 at 18:08

score 0 · Answer 2 · answered Mar 23 '20 at 17:38


df = pd.DataFrame({'word':wordlist, 'count':countlist})

df.columns=['technology word','count']
df=df.set_index('technology word')
df.drop_duplicates()
print(df)
                 count
word                  
steam               84
heating              9
horizontal well      4
electromagnetic      2
single well          1
steam               89
foam                 2
heating             10
horizontal well      4
solvent              5
hexane               7
electromagnetic      2
steam foam           1
surfactant         106
single well          1
miscible             1

Well the loop is updating the count correctly, but I need to print only the final words and counts. I tried yours but it doesn't work and I don't want it deleting the word steam with highest count 89.

score 0 · Answer 3 · answered Mar 23 '20 at 18:20

                 count
word                  
steam               84
heating              9
horizontal well      4
electromagnetic      2
single well          1
steam                5
foam                 2
heating              1
solvent              5
hexane               7
steam foam           1
surfactant         106
miscible             1
```
How can I add the count of words like example steam=84+5=89 from this data frame. How can I have the code uniquely count the word and add the number of occurences to one word.

Counting the words uniquely in a dataframe

3 Answers3