How to count values since the first non nan value?

Question

I have this df:

         CODE    TMAX
0        000130  NaN
1        000130  NaN
2        000130  32.0
3        000130  32.2
4        000130  31.1
5        158328  22.5
6        158328  8.8
7        158328  NaN
8        158328  NaN
9        158328  9.2
...      ...     ...

I want to count the number of non nan values and the number of nan values in the 'TMAX' column. But i want to count since the first non NaN value.

Expected result: 6 non nan values and 2 NaN values.

How can i do this?

Thanks in advance.

Please do not change the question once answered. If you have another question that is different from the one you are asking, please ask another question or do some research on your own. It will make people visiting your question confused as the question and answer differ. @Javier — Vishnudev Krishnadas, Jun 18 '21 at 06:19

jezrael · Answer 1 · 2021-06-18T06:25:11.133

1

Use Series.notna with Series.cummax for filter out first NaNs and then count by Series.value_counts with rename index values by dict:

m = df.TMAX.notna()
s = m[m.cummax()].value_counts().rename({True:'non NaNs',False:'NaNs'})
print (s)
non NaNs    6
NaNs        2
Name: TMAX, dtype: int64

edited Jun 18 '21 at 06:25

answered Jun 18 '21 at 05:45

jezrael

822,522
95
1,334
1,252

Mustafa Aydın · Answer 2 · 2021-06-18T06:01:21.170

1

result = (df.TMAX
            .truncate(before=df.TMAX.first_valid_index())
            .mask(pd.Series.notna, other="non-NaN")
            .value_counts(dropna=False))

where we truncate the series before its first valid index i.e., first non-Nan index, and then turn it into a NaN or not NaN series with mask. Then we count the values (including NaNs),

to get

>>> result

non-NaN    6
NaN        2

edited Jun 18 '21 at 06:01

answered Jun 18 '21 at 05:48

Mustafa Aydın

17,645
4
15
38

Vishnudev Krishnadas · Accepted Answer · 2021-06-18T06:28:01.170

1

Use first_valid_index to find the first non-NaN index and filter. Then use isna to create a boolean mask and count the values.

output = (
    df.loc[df.TMAX.first_valid_index():, 'TMAX']
    .isna()
    .value_counts()
    .rename({True: 'NaN', False: 'notNaN'})
)

Output

notNaN    6
NaN       2
Name: TMAX, dtype: int64

edited Jun 18 '21 at 06:28

answered Jun 18 '21 at 05:53

Vishnudev Krishnadas

10,679
2
23
55

How to count values since the first non nan value?

3 Answers3

Linked