0

I try to lemmatize the column "tokenized" in a dataframe. One cell of the column "tokenized" looks as follows " yeah simply zurich generic serving think media bland prepared curry kind paying well loves used parboiled oily place elaborate non tasteful stay underspiced institution vegetarian indian clueless away hiltl anyone served support veg long like normal strong worth insult not rice kitchen know wont food cuisine fantastic fan time term patrons ".

When I run my code it returns something like this: ",,e,n,d,e,d,,,p,a,y,i" which is not what i want. How can I lemmatize full words?

This is my code:

reviews_english['tokenized_lem'] = reviews_english['tokenized'].apply(
                    lambda lst:[lmtzr.lemmatize(word) for word in lst])
reviews_english
Davide Fiocco
  • 5,350
  • 5
  • 35
  • 72
Nicole
  • 5
  • 2

1 Answers1

0

The problem is that your "tokenized" column doesn't look ready to apply the lemmatization step, as it contains a string, not a list of tokens. In other words, instead of having

" yeah simply zurich generic serving ..."

you should have in your dataframe tokenized cell a list of tokens (generated with a tokenizer from your initial sentence), as in

["yeah", "simply", "zurich", "generic", "serving", ...]

If you don't have a proper list of tokens in your dataframe cell, python will iterate in your apply/lambda list comprehension character by character, which is clearly not what you want.

Davide Fiocco
  • 5,350
  • 5
  • 35
  • 72
  • For an example using spaCy to do the task, see also https://stackoverflow.com/questions/47498293/how-can-i-lemmatize-strings-in-pandas-dataframes/49322360#49322360 – Davide Fiocco Jun 26 '21 at 07:57