0

I have a lexicon dictionary in this shape 

6   ابن جزمه    1
7   ابو جهل -1
8   اتق الله    -1
9   اتقو الله   1

I want to create a new list containing the score of each sentence based on the lexicon adding the score of each word and if no words exist append zero when I implement my code I get len(lex_score) = 3679 after I add elif condition I get len(lex_score) = 95079

the len(lex_score) should equal 6064

lex_score = []
def lexic(text):
    for tweet in sentences:
        score = 0
        for word in tweet.split():
            if word in lexicon:
                score = score+lexicon[word]
            elif word not in lexicon:
                score = 0
                lex_score.append(score)

I want to create a new column in the data frame containing the score of each sentence. what am I doing wrong? and is there a better way to do so ?

Muhammed Eltabakh
  • 375
  • 1
  • 10
  • 24
  • Can you show your `lexicon` dictionary in python format? Also, show more code please. E.g. give an example of `sentences`. – Bill Sep 03 '17 at 20:58
  • Note, you can replace the `elif` condition with `else:` if there are no other possible outcomes. – Bill Sep 03 '17 at 21:01

1 Answers1

1

IIUC, you can just sum the scores of valid lexicon entries in each tweet, and then append that score to lex_score on each iteration of sentences.

Note: I'm assuming text == sentences - otherwise there's a missing line where text is broken down into sentences. Either way, this basic approach should still work:

def lexic(text):
    lex_score = []
    for tweet in text: # assuming sentences == text
        score = sum([lexicon[word] for word in tweet.split() if word in lexicon])
        lex_score.append(score)
    return lex_score
andrew_reece
  • 20,390
  • 3
  • 33
  • 58
  • Thanks a lot it worked. any Idea If I have another list with negation words how do I handle it with that lexicon ? – Muhammed Eltabakh Sep 04 '17 at 10:45
  • You're welcome! I'm not sure I understand your negation words question, happy to help if I can. If it's different enough from your original question, consider opening a separate question. If this answer resolved your original question, please mark it accepted by clicking the check mark to the left of the answer. – andrew_reece Sep 04 '17 at 15:05
  • I have a list of negation words and I'm worried they will affect the words in lexicon so I want to take it in my consideration. how can I do so ? – Muhammed Eltabakh Sep 04 '17 at 15:17
  • Can you update your original post with an example, including the impact you're concerned about? – andrew_reece Sep 04 '17 at 15:41