3

I have a pandas dataframe:

>>> X_df.shape
Out[35]: (177, 2762)
>>> X_df.ix[0:5,1000:1005]
Out[40]: 
              1000    1001    1002    1003    1004    1005
2016-01-04 119.225     nan     nan     nan     nan     nan
2016-01-05 119.225 119.189 119.177 119.160 119.203 119.220
2016-01-06 119.175 119.175     nan 119.204 119.208 119.221
2016-01-07 118.532     nan 118.542 118.529     nan 118.534
2016-01-08 117.861 117.820 117.762     nan 117.686 117.709

I create the mean of first differences along each row:

>>> mu = (X_df.diff(1,axis=1)).mean(axis=1)
>>> mu.head()
Out[42]: 
2016-01-04   -0.001
2016-01-05   -0.001
2016-01-06   -0.000
2016-01-07   -0.000
2016-01-08   -0.000
dtype: float64

Then if i try to subtract this mean from each value, I get all nans:

>>> (X_df.diff(1,axis=1)-mu).ix[0:5,1000:1005]
Out[51]: 
            1000  1001  1002  1003  1004
2016-01-04   nan   nan   nan   nan   nan
2016-01-05   nan   nan   nan   nan   nan
2016-01-06   nan   nan   nan   nan   nan
2016-01-07   nan   nan   nan   nan   nan
2016-01-08   nan   nan   nan   nan   nan

different way of subtraction

>>> ((X_df.diff(1,axis=1).subtract(mu,axis=1))).ix[0:5,1000:1005]
Out[52]: 
            1000  1001  1002  1003  1004
2016-01-04   nan   nan   nan   nan   nan
2016-01-05   nan   nan   nan   nan   nan
2016-01-06   nan   nan   nan   nan   nan
2016-01-07   nan   nan   nan   nan   nan
2016-01-08   nan   nan   nan   nan   nan

I get same results on using mu[:,] instead of mu

Is there a way to fix this?

dayum
  • 1,073
  • 15
  • 31

1 Answers1

3

You need sub with axis=0:

a = (X_df.diff(1,axis=1))
print (a)
            1000   1001   1002   1003   1004   1005
2016-01-04   NaN    NaN    NaN    NaN    NaN    NaN
2016-01-05   NaN -0.036 -0.012 -0.017  0.043  0.017
2016-01-06   NaN  0.000    NaN    NaN  0.004  0.013
2016-01-07   NaN    NaN    NaN -0.013    NaN    NaN
2016-01-08   NaN -0.041 -0.058    NaN    NaN  0.023

print (a.sub(a.mean(axis=1), axis=0))
#sub same as subtract, only less typing
#print (a.subtract(a.mean(axis=1), axis=0))
            1000      1001      1002   1003      1004      1005
2016-01-04   NaN       NaN       NaN    NaN       NaN       NaN
2016-01-05   NaN -0.035000 -0.011000 -0.016  0.044000  0.018000
2016-01-06   NaN -0.005667       NaN    NaN -0.001667  0.007333
2016-01-07   NaN       NaN       NaN  0.000       NaN       NaN
2016-01-08   NaN -0.015667 -0.032667    NaN       NaN  0.048333
jezrael
  • 822,522
  • 95
  • 1,334
  • 1,252
  • yes i noticed that too..but dont understand why. Isn't axis=0 meant for columns? – dayum Mar 02 '17 at 06:14
  • No, it is for index. for columns is default values `axis=1` – jezrael Mar 02 '17 at 06:15
  • 1
    Maybe help [this](http://pandas.pydata.org/pandas-docs/stable/basics.html#matching-broadcasting-behavior) – jezrael Mar 02 '17 at 06:17
  • If i do DF.mean(axis=1) , it takes mean along each index. By similar logic, DF.sub(x, axis=1) should subtract along each index, right? – dayum Mar 02 '17 at 06:18
  • I understand your logic, but ither is about matching by `index` (axis=0) aor by columns `index=1`. – jezrael Mar 02 '17 at 06:20