0

Example:

DATE ENERGY_INDEX
0 01/1939 3.3842
1 02/1939 3.4100
2 03/1939 3.4875
3 04/1939 3.5133
4 05/1939 3.5133

How to check whether data for any month is being missed in the timeseries data?

My approach to find the missing values:

df['DATE']=pd.to_datetime(df['DATE'],format='%m/%Y')
df.index=df['DATE']
df['DATE'].max()-df['DATE'].min()` - output - Timedelta('29463 days 00:00:00')
df.shape - output - (969,2)

Actually, df['DATE']=pd.to_datetime(df['DATE'],format='%m/%Y') is adding the date parameter in the date.

Neuron
  • 5,141
  • 5
  • 38
  • 59
Amit Karmakar
  • 11
  • 2

1 Answers1

0

I suggest you to use Pandas dataframe :

!pip install pandas
import pandas as pd
main_na = pd.Dataframe(your_datas) #Where your_datas is your list or np array
main_na = main_df.notna()
for col in main_na.columns:
    diff = len(main_na[col]) - main_na.count()[col]
    if diff > 0:
        print( f"{len(main_na[col]) - main_na.count()[col]} NaN found in column {col}")
main_df.ffill(inplace=True)
main_df.dropna()

This code, that I often use, prints the number of NaN in your data

But if you are in a case where you might have missing rows you should use your date column to create a new timestamp column (take a look here : Python pandas convert datetime to timestamp effectively through dt accessor ). And then set this new column as the index of your dataframe using :

main_df.set_index("name of your new timestamp colums")

and do this :

main_df.sort_index(ascending=True, inplace=True)
time_df = main_df.copy()
time_df["diff"] = time_df.index
time_df["diff"] = time_df["diff"]-time_df["diff"].shift(1)
time_df = time_df["diff"]
print(time_df.value_counts())

It will display the timestamp difference between each of your rows like this for exemple :

900000.0      68383
4500000.0         3
8100000.0         2
9900000.0         2
17100000.0        2
21600000.0        1
13500000.0        1
14400000.0        1
5400000.0         1
6300000.0         1
Name: diff, dtype: int64

With the time difference as the left column and the number of case as the right column.

Clément Perroud
  • 483
  • 3
  • 12