Case Study : The Spock Conspiracy Trial, in SAS - by using proc glm & the Bonferroni Correction

Case Study: The Spock Conspiracy Trial in SAS
Main question is there is a difference among the 6 other judges?
We will use multiple linear regression model with 6 dummy predictor variables. (One-way ANOVA)
Reference: http://www.inside-r.org/node/159733 


[1] The Hypothesis and Model 
 $H_{0}=\mu_A = \mu_B= ...=\mu_E = \mu_F$
 $H_{1}$ = At least one judge is different from others.
 Model : $Y_{i}=\beta_0 + \beta_{1} \cdot I_{A_{,i}} + \beta_{2} \cdot I_{B_{,i}} + ... + \beta_{5} \cdot I_{F_{,i}} + e_{i}$, where I is an indicator variable.

[2] SAS Code

Please refer to the previous post how to read the data using infile statement. The data was divided into two groups; 'SPOCKS' and 'OTHERS' using if statement.  This data name is 'all', and we will use this 'all' data.

Read the 'all' data, and we will exclude 'SPOCK' group using ne statement as our purpose is to find a difference among 6 other judges. So we create indicator variables. This data name is 'otherjudges'.

We use proc glm statement to get One-way ANOVA result with Bonferroni correction. 

* cldiff statement reference: https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glm_sect018.htm
*pdiff statement reference : http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_glm_sect016.htm


[3.1] SAS Result - Overall AVNOA

As there are 6 judges, so the number of degrees of freedom is 5. The F-value (the ratio of variance) is 1.22 = 54.291574/53.59180. The P-value is 0.3239 which is higher than 0.05, so we can see that there is no evidence of difference among the means of the 6 judges. (We fail to reject the null hypothesis). We don't need to see the post-hoc pairwise comparison, but let's see the result of the LSmeans with the Bonferroni adjustment.


[3.2] SAS Result - LSMeans and Difference Matrix

The least square means are following below. From the ANOVA result above, there is no difference among those means.  

In the difference matrix below, the null hypothesis is that ith LS means is equal to jth LS mean. As all p-values in the matrix are higher than 0.05, so we fail to reject the null hypothesis.

[3.3] SAS Result - Difference Matrix with the Bonferroni Correction
How to read the difference matrix is the same above. The null hypothesis is that ith LS means is equal to jth LS mean. With the adjustment for multiple comparison, all the p-values are 1 which is higher than 0.05. Therefore, we have a strong evidence that there is no difference in means among 6 judges.

 

[Multiple Comparisons] The Bonferroni Method

 

If we need multiple tests for comparing means with a single data set, there are many pairwise comparisons. For example, when comparing three means(A, B, and C), there are 3 pairwise comparisons, T1=(A,B), T2=(B,C) and T3=(C,A). If there are G groups means, there will be $k=\binom{G}{2}= \frac{G \cdot (G-1)}{2}$ pairwise tests which are T1,...,Tk. 

 

So based on the Bonferroni inequality, $P(A\cup B)\leq P(A)+P(B)$,

P(T1 incurs Type I error) + ... + P(Tk incurs Type I error) = $P(\cup_{i=1}^k A_{i})\leq \sum_{i=1}^k P(A_{i})$

 

For example, let $\alpha$ =P(Type I error)=0.05 and G = # of means = 7, then k=(7*6)/2=21,

then P(at least 1 Type I error) = $1-(1-0.05)^{21}=0.6594$, which means there is an increased chance of making at least one Type I error rate.

P(at least 1 Type I error among independent k tests) = 1 - P(No Type I error among k tests).

 

In order to control to have less than $\alpha$, each of k pairwise tests is done at level $\frac{\alpha}{k}$!! In this example, our adjusted error rate will be 0.05/21=0.0024!! So the probability of Type I error rate in each test should be 0.0024 so that the probability of Type I error rate among 21 pairwise test will be similar to 0.05.
P(at least I type I error) = $1-(1-0.0024)^{21}=0.0492 \approx 0.05$
  

So this Bonferroni method is conservative as we are asking greater evidence which means overall Type I error rate is much less than $\alpha$.