3

Being a regular R user, I am learning to do analysis using python, I started with chi-square and did the following:

R

> chisq.test(matrix(c(10,20,30,40),nrow = 2))$p.value               # test1
[1] 0.5040359
> chisq.test(matrix(c(1,2,3,4),nrow = 2))$p.value                   # test2
[1] 1
Warning message:
In chisq.test(matrix(c(1, 2, 3, 4), nrow = 2)) :
  Chi-squared approximation may be incorrect
> chisq.test(matrix(c(1,2,3,4),nrow = 2),correct = FALSE)$p.value   # test3
[1] 0.7781597
Warning message:
In chisq.test(matrix(c(1, 2, 3, 4), nrow = 2), correct = FALSE) :
  Chi-squared approximation may be incorrect

Python

In [31]:
temp = scipy.stats.chi2_contingency(np.array([[10, 20], [30, 40]]))  # test1
temp[1] # pvalue
Out[31]:
0.50403586645250464
In [30]:
temp = scipy.stats.chi2_contingency(np.array([[1, 2], [3, 4]]))      # test2
temp[1] # pvalue
Out[30]:
0.67260381744151676

For test1, I am satisfied because tests from python and R show similar result, but test2 is not the case, since R have the parameter correct, so I changed it from default, and the p-value generated is not the same.

Is there anything wrong in my code? Which one should I "believe"?

update 01

Thanks for the feedback. I am aware that chi square test should not be used for cell with value smaller than 5, and I should use fisher exact test instead, my concern is why R and Python gives p-value with such a huge difference.

lokheart
  • 23,743
  • 39
  • 98
  • 169
  • Really you should believe neither, since Chi-squared is not valid for such small samples. See [this question](http://stackoverflow.com/questions/9330114/chi-squared-test-in-python). – BrenBarn Jul 30 '14 at 04:46
  • 2
    Believe neither of them. from the Python documentation: "An often quoted guideline for the validity of this calculation is that the test should be used only if the observed and expected frequency in each cell is at least 5." – JeremyS Jul 30 '14 at 04:46
  • 1
    You could always calculate using the [wiki definition](http://en.wikipedia.org/wiki/Pearson's_chi-squared_test) and see which matches. In R you can verify these calculations with `m <- matrix(1:4, nrow=2); e <- t(t(rowSums(m))) %*% t(colSums(m))/sum(m); xq <- sum((e-m)^2/e); 1-pchisq(xq,1)` (Of course as already pointed out, this is not a good test to use with small counts). It may be more helpful to return *all* information rather than just p-value (ie deg-of-freedom, test statistic etc). – MrFlick Jul 30 '14 at 04:57

1 Answers1

5

Aside from the cell count < 5 issue, in my experience both R and Python implementations of statistical tests often have various corrections (that are supposed to improve on the basic method) enabled by default. Turning the correction off seems to make the scipy p value match R's:

scipy.stats.chi2_contingency(np.array([[1, 2], [3, 4]]), correction=False)

Out[6]: 
# p-val = 0.778159
(0.079365079365079388, 0.77815968617616582, 1, array([[ 1.2,  1.8],
        [ 2.8,  4.2]]))

The same applies for t-tests etc., where the default may or may not be to assume equal variances. Basically whenever you're having trouble matching output between statistical software, start looking at the default arguments to see if you should be enabling or disabling these adjustements.

Marius
  • 58,213
  • 16
  • 107
  • 105