pandas - add missing rows on the basis of column values to have linspace

Question

I have a pandas dataframe like

     a   b  c
0  0.5  10  7
1  1.0   6  6
2  2.0   1  7
3  2.5   6 -5
4  3.5   9  7

and I would like to fill the missing columns with respect to the column 'a' on the basis of a certain step. In this case, given the step of 0.5, I would like to fill the 'a' column with the missing values, that is 1.5 and 3.0, and set the other columns to null, in order to obtain the following result.

     a     b    c
0  0.5  10.0  7.0
1  1.0   6.0  6.0
2  1.5   NaN  NaN
3  2.0   1.0  7.0
4  2.5   6.0 -5.0
5  3.0   NaN  NaN
6  3.5   9.0  7.0

Which is the cleanest way to do this with pandas or other libraries like numpy or scipy?

Thanks!

score 5 · Accepted Answer · answered Feb 27 '19 at 12:52

Create array by numpy.arange, then create index by set_index and last reindex with reset_index:

step= .5
idx = np.arange(df['a'].min(), df['a'].max() + step, step)
df = df.set_index('a').reindex(idx).reset_index()
print (df)
     a     b    c
0  0.5  10.0  7.0
1  1.0   6.0  6.0
2  1.5   NaN  NaN
3  2.0   1.0  7.0
4  2.5   6.0 -5.0
5  3.0   NaN  NaN
6  3.5   9.0  7.0

score 5 · Answer 2 · answered Feb 27 '19 at 12:58

One simple way to achieve that is to first create the index you want and then merge the remaining of the information on it:

import pandas as pd
import numpy as np

df = pd.DataFrame({'a': [0.5, 1, 2, 2.5, 3.5],
                   'b': [10, 6, 1, 6, 9],
                   'c': [7, 6, 7, -5, 7]})
ls = np.arange(df.a.min(), df.a.max(), 0.5)
new_df = pd.DataFrame({'a':ls})
new_df = new_df.merge(df, on='a', how='left')

pandas - add missing rows on the basis of column values to have linspace

2 Answers2