Why does the filtering by notnull() not work?

Question

I have a DataFrame cMean. Its origin is some resampling of some data. It contains many NaN values and I wanted to get rid of them so I tried cMean[cMean.notnull()]. However, they still show up:

Can you explain what is going on here? It seems cMean.notnull() works correct, as you can see here:

mozway · Accepted Answer · 2023-01-13T14:07:37.850

2

This doesn't work as cMean.notnull() gives you a 2D output, and you need a 1D Series to perform boolean indexing.

Use the specific column of interest:

out = cMean[cMean['population'].notnull()]

Or aggregate with any/all if you want to consider several columns:

out = cMean[cMean.notnull().any(axis=1)]

Note that you can also use dropna:

out = cMean.dropna(subset=['population'])

edited Jan 13 '23 at 14:07

answered Jan 13 '23 at 14:02

mozway

194,879
13
39
75

Thanks, that works. Can you explain what was wrong with my approach? – principal-ideal-domain Jan 13 '23 at 14:03
1

I updated the answer, using `cMean.notnull()` gives a 2D output, you need 1D to slice ;) – mozway Jan 13 '23 at 14:04

Why does the filtering by notnull() not work?

1 Answers1