0

I have two distributions below (Kaggle dataset: Rossman sales) that look similar visually: Sales on normal days & sales on school holiday.

enter image description here

However, they seems to fail z-test (hypothesis testing) in Python - why is that so?

How should I perform the statistical test (z-test) in Python? Should I use pooled or unequalvar (should I use same variance or different)? I also found out that switching School_hol_sales and Normal_day_sales in the code below yield different results and I am not sure why.

School_hol_sales = df[(df.Open==1)&(df.SchoolHoliday==1)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
Normal_day_sales = df[(df.Open==1)&(df.SchoolHoliday==0)&(df.StateHoliday=='0')&(df.Promo==0)].Sales
School_hol_sales.mean(), Normal_day_sales.mean() # (6230.4, 5904.6)
School_hol_sales.std(), Normal_day_sales.std() # (2841.8, 2602.9)


# which is the correct one?
import statsmodels.stats.api as sms

cm = sms.CompareMeans(sms.DescrStatsW(School_hol_sales), sms.DescrStatsW(Normal_day_sales))
z, pval = cm.ztest_ind(alternative='larger', usevar='unequal')
print('z: {} , pval: {}'.format(z, pval))


from statsmodels.stats.weightstats import ztest

z, pval = ztest(School_hol_sales,Normal_day_sales, alternative='larger', usevar='pooled', ddof=1.0)
print('z: {} , pval: {}'.format(z, pval))

Output: 

z: 28.53350149055591 , pval: 2.2504631945823565e-179
z: 30.17089944207645 , pval: 2.853425122518376e-200
jasonlcy91
  • 454
  • 6
  • 14
  • If I may ask you, what exactly do you want to test? Is it the possibility that the two samples you mention are actually samples of the same population and thus have the same distribution? – DavidPM Nov 28 '18 at 16:34
  • 1
    Statistical significance is not the same thing as practical significance. If two means are actually different, in any degree no matter how small, a significance test will almost certainly fail given sufficiently large samples; this is a well-known feature or bug, depending on one's point of view. My advice is dump all significance tests and work only with practical significance, i.e. assess a value (in money, time, resources, whatever) for your actions and go from there. – Robert Dodier Nov 28 '18 at 19:28

0 Answers0