[Multiple Comparisons] The Bonferroni Method

 

If we need multiple tests for comparing means with a single data set, there are many pairwise comparisons. For example, when comparing three means(A, B, and C), there are 3 pairwise comparisons, T1=(A,B), T2=(B,C) and T3=(C,A). If there are G groups means, there will be $k=\binom{G}{2}= \frac{G \cdot (G-1)}{2}$ pairwise tests which are T1,...,Tk. 

 

So based on the Bonferroni inequality, $P(A\cup B)\leq P(A)+P(B)$,

P(T1 incurs Type I error) + ... + P(Tk incurs Type I error) = $P(\cup_{i=1}^k A_{i})\leq \sum_{i=1}^k P(A_{i})$

 

For example, let $\alpha$ =P(Type I error)=0.05 and G = # of means = 7, then k=(7*6)/2=21,

then P(at least 1 Type I error) = $1-(1-0.05)^{21}=0.6594$, which means there is an increased chance of making at least one Type I error rate.

P(at least 1 Type I error among independent k tests) = 1 - P(No Type I error among k tests).

 

In order to control to have less than $\alpha$, each of k pairwise tests is done at level $\frac{\alpha}{k}$!! In this example, our adjusted error rate will be 0.05/21=0.0024!! So the probability of Type I error rate in each test should be 0.0024 so that the probability of Type I error rate among 21 pairwise test will be similar to 0.05.
P(at least I type I error) = $1-(1-0.0024)^{21}=0.0492 \approx 0.05$
  

So this Bonferroni method is conservative as we are asking greater evidence which means overall Type I error rate is much less than $\alpha$.