-2

I have a dataframe whose values are lists which contains 'nan'. Is there an easy and pythonic way to remove those 'nan' values from lists within the dataframe? I have defined a function which returns a list without 'nan', but how can I apply it to dataframe inplace?

def remove_nan(input_list):
    temp_list = [x for x in input_list if x!='nan']
    return temp_list

test = ['nan', 'nan', 'SHM System', 'nan', 'nan', 'nan']

remove_nan(test)
['SHM System']

This function works on individual list and returns clean list as shown in the output above. How can I apply this function, or if there is a better way, to remove all 'nan' values from lists within a dataframe? I have tried applymap and apply but didn`t work.

df_combined.applymap(remove_nan)
Gibrail Hassan
  • 85
  • 7
  • 1
    Show your bit of code that didn't work too... [ask] [mre] – Julien Mar 17 '21 at 04:32
  • This is what I did: df_combined.applymap(remove_nan) It didn`t change the dataframe. – Gibrail Hassan Mar 17 '21 at 04:35
  • 1
    `df.applymap(lambda x: [*filter(pd.notna, x)])` – piRSquared Mar 17 '21 at 04:37
  • Did you try `df_combined = df_combined.applymap(remove_nan)` ? The applymap method doesn't modify the original DataFrame unless you set it equal to what it's returning – Derek O Mar 17 '21 at 04:37
  • @DerekO Yeah, I can see the copy of this data frame, it is same. Hasn`t changed. – Gibrail Hassan Mar 17 '21 at 04:40
  • I tried with an example DataFrame using your variable `test`, and it seems to work: `df = pd.DataFrame({'a':[test]*3,'b':[test]*3})` to create a DataFrame with lists containing 'nan' as you described, then `df = df.applymap(remove_nan)` modifies df so that none of the entries contain the string `nan` – Derek O Mar 17 '21 at 04:43
  • Yeah, this example also works for me, but same code doesnt work on the dataframe I`m trying to clean. However, solution from @piRSquared did work. Thank you for the help. :) – Gibrail Hassan Mar 17 '21 at 04:54

1 Answers1

2

The following line of code worked for me. Thanks to @piRSquared.

df.applymap(lambda x: [*filter(pd.notna, x)])
Gibrail Hassan
  • 85
  • 7