0

Given the definition of standard deviation as:
the square root of the expectation of X-squared minus the expectation of X, squared;
why do the following two calculations of exponentially weighted standard deviation not result in the same values?

import numpy
import pandas
x = pandas.Series(numpy.random.default_rng(0).normal(size=1000))
com = 2
a = x.ewm(com).std()
b = numpy.sqrt((x**2).ewm(com).mean() - (x.ewm(com).mean())**2)

pandas.concat([a,b], axis=1).tail(5)

Output:

a b
995 1.535113 1.373047
996 1.302008 1.164551
997 1.320120 1.180751
998 1.215099 1.086818
999 1.002668 0.896814
handlist
  • 197
  • 1
  • 9
  • 1
    FYI, `a/b` is a constant. – Quang Hoang Sep 02 '22 at 18:43
  • @QuangHoang, interesting, thank you. Are you able to describe why this is? – handlist Sep 02 '22 at 18:46
  • 1
    Just digged into the code and realized that it's due to [`bias=False`](https://en.wikipedia.org/wiki/Standard_deviation#Unbiased_sample_standard_deviation) as default. Passing `bias=True` to `std(bias=True)` gives the same answer as `b` – Quang Hoang Sep 02 '22 at 18:48
  • Yes thank you, that makes `a` like `b`. But how should I modify the calculation in `b` to make it like `a`? How do I bias correct the calculation in `b`? – handlist Sep 02 '22 at 18:52
  • 1
    @SergeyBushmanov Most ewm's methods are native C. I did look into [github code](https://github.com/pandas-dev/pandas/blob/2f8d0a36703e81e4dca52ca9fe4f58c910c1b304/pandas/core/window/ewm.py), which is somewhat equivalent to the signature. – Quang Hoang Sep 02 '22 at 18:55
  • 1
    @SergeyBushmanov I don't, hence the comments. I do think that there is a bias coefficient, just that I don't know what it is or how to calculate it. – Quang Hoang Sep 02 '22 at 19:07
  • 1
    @handlist [this answer](https://stackoverflow.com/a/50236294/4238408) seems to show the bias coefficient. You can check it out. – Quang Hoang Sep 02 '22 at 19:12
  • @QuangHoang Thank you very much! From that link I was able to derive the bias correction term of sqrt(1+(2*com)) – handlist Sep 02 '22 at 19:57

0 Answers0