Python pandas .isnull() does not work on NaT in object dtype

Question

I have this series:

ser=pd.Series([11,22,33,np.nan,np.datetime64('nat')],name='my_series')

The series looks like this:

0     11
1     22
2     33
3    NaN
4    NaN
Name: my_series, dtype: object

But I get only one True for NULL values:

ser.isnull()

0    False
1    False
2    False
3     True
4    False
Name: my_series, dtype: bool

Is it a bug or how can I count correctly the NULL values in a pandas series? This does not help:

ser=ser.replace('NaN',np.nan)

Thanks!

does `ser.loc[4]` result in `NaN` or `NaT`? I think that datetime64 can deal interchangeably with NaN and NaT, but attributing a NaT to a series of 'object' causes the problem — vmg, Sep 30 '15 at 10:42
Yes. Of course this is only an example... The second `NaN` has been converted from `NaT` (??) but there can be some trouble during conversion. Or maybe there is no conversion:`ser.loc[4]` results `NaT` — ragesz, Sep 30 '15 at 10:47
what is your pandas version? Do you get same issue if you use `pd.NaT` instead of `np.datetime64('nat')` ? — Anand S Kumar, Sep 30 '15 at 10:58
Oh, `.isnull()` works perfectly with `pd.NaT`. Thanks!!! pandas version is 0.16.2, numpy version is 1.9.2 — ragesz, Sep 30 '15 at 11:01
should prob work, issue created [here](https://github.com/pydata/pandas/issues/11206) — Jeff, Sep 30 '15 at 11:21
This looks to be due to an outdated version of pandas or numpy. Works fine for me (pd 0.23.4, np 1.15.4) `np.isnat(np.datetime64('nat')) == True`, `pd.isnull(np.datetime64('nat')) == True` — gosuto, Dec 29 '18 at 08:51

score 6 · Answer 1 · answered Apr 09 '19 at 12:35

6

I ran into a similar issue this morning but in an str series! This worked for me and with your sample data as well:

pd.isna(ser)

answered Apr 09 '19 at 12:35

nck

score 0 · Answer 2 · answered Oct 01 '15 at 23:31

0

To get around this, you can also do

series.apply(lambda x: str(x) == "nat")

Then you can still use np.datetime if you want

answered Oct 01 '15 at 23:31

mirthbottle

2

This does not work. `series.apply(lambda x: str(x) == "NaT")` **does** work. Beware case sensitivity in string comparisons. – C8H10N4O2 Dec 07 '16 at 21:41

score 0 · Answer 3 · answered Aug 15 '18 at 04:28

Surprisingly I got following output after execution the code you given:

calculate length of series object and subtraction will give you the count for null values. As follows:**

len(ser)-(ser[ser.isnull()==False]).count()

3 Answers3