A Problem with Family-wise Error Calculation

Question

Consider I have a data set with a 12 variable and it contains about 100 cases. If I am going to do a t-test between two variables, then doing an ANOVA between 3 other variables then a post-hoc. and maybe after that a Mann-Whitney for non-parametric variable... my question is if I am going to use a Bonferroni for the Post-hoc in ANOVA, how I should calculate the family-wise error? when does the family of analyses start? when does it end?

Answer 1

As the name suggest, FWER is the probability of making one or more false discoveries, when performing multiple hypotheses tests.

If you try two different test for the same hypothesis, it is the same hypothesis, so you don't do FWER.

In the post-hoc, you are repeating the test for differences between variables and you can apply the Bonferroni there.

Answer 2

"If I am going to do a t-test between two variables, then doing an ANOVA between 3 other variables then a post-hoc" seems misleading.

If you're talking about 1-way ANOVA, then you can do it for 2 variables only: 1 numeric/metric response variable (researchers usually call it 'dependent variable') and 1 categorical/non-metric predictor variable (researchers usually call it 'independent variable') with 2 or more categories.

If the Null hypothesis (Mean of all groups/categories are same) of ANOVA is rejected, it means at least one of the group/category is having a significantly different mean.

To further analyze which of the groups are having statistically significantly different means, you need pair-wise comparisons. For that, you can use Post-Hoc Tests.

Post hoc tests consist of pairwise comparisons that are designed to compare the mean of all different combinations of the groups. So, it is rather like taking every pair of groups and then performing a t-test on each pair of groups.

To adjust for family-wise errors, there are some ways. Bonferroni suggests a simple way of dividing the p-value by the number of tests, known as Bonferroni adjustment for p-value. However, this adjustment is very conservative. In order to decrease false discoveries, it increases the chances of false negatives. There are some other posthoc tests too like LSD, Tuckey, SNK, Welch Q process, Dunn and Scheffe process, etc.

Now, coming to your question: "how I should calculate the family-wise error? when does the family of analyses start? when does it end?"

Imagine you perform separate t-tests for each pair of groups. You will need to make three comparisons between means (of response variable) of group A & B, A & C, and B & C_assuming that your predictor/independent variable has three groups A, B and C.

If each of these t-tests uses a .05 level of significance then for each test the probability of falsely rejecting the null hypothesis (known as a Type I error) is only 5%. Therefore, the probability of no Type I errors is .95 (95%) for each test.

If we assume that each test is independent (hence, we can multiply the probabilities) then the overall probability of no Type I errors is (.95)3 = .95 × .95 × .95 = .857, because the probability of no Type I errors is .95 for each test and there are three tests.

Given that the probability of no Type I errors is .857, then we can calculate the probability of making at least one Type I error by subtracting this number from 1 (remember that the maximum probability of any event occurring is 1). So, the probability of at least one Type I error is 1 − .857 = .143, or 14.3%.

Therefore, across this group of tests, the probability of making a Type I error has increased from 5% to 14.3% This error rate across statistical tests conducted on the same experimental data is known as the familywise or experiment-wise error rate.

Read More about selecting an appropriate test, assumptions & interpretation, multiple tests, posthoc, and contrast. Whatever tool (SPSS/SAS/R) you're using, the concept remains the same. You may refer Discovering Statistics using IBM SPSS or Discovering Statistics usin R by Andy Field for more details.