Idea Transcript
1 MULTIPLE COMPARISONS 1. Bonferroni Method. We have seen: If we form two 95%confidence intervals for two means or two effect differences, etc., then the probability that, under repeated sampling with the same design, the procedures used will give intervals each containing the true mean, effect differences, etc. might only be 90% -- we have no reason to believe it must be any higher without any more information. Thus the simultaneous or family-wise or overall confidence level is 90% Similarly, if we are forming m confidence intervals, each with confidence level 1- α individually, then analogous probability calculations show that the simultaneous or family-wise or overall confidence level will be only 1-mα . This gives us one method of deciding what individual confidence levels to choose if we want a given overall confidence level: If we want overall level 1- α, then choose individual level 1-α/m. For example, if we are forming 5 confidence intervals and want an overall 95% confidence level, then we need to use the procedure for individual 99% confidence intervals. This method of forming simultaneous confidence intervals is called the Bonferroni method. It gives wide intervals. Example: In the battery experiment, the individual 95% confidence intervals for the four mean shown in the Minitab output have a Bonferroni overall confidence level 80%. If we want an overall confidence level 95% for the four confidence intervals, we need to calculate individual 98.75% confidence intervals. These would have standard error 2368 msE = = 24.33 and use t-value t(12, .99375) = 2.9345, giving confidence 4 ri intervals of half-width 71.40, in contrast to the half-width 24.33x2.1254 = 51.71 for the individual 95% confidence intervals -- more than a third as wide. The phenomenon shown in the example is typical: To get a certain family confidence level, you will get wider confidence intervals that those formed with the individual confidence level. A Bonferrroni approach can also be used for hypothesis tests: If you want to do m hypothesis tests on your data, and you want an overall type I error rate of α (that is, you want to have probability of falsely rejecting at least one of the null hypotheses less than α), you can achieve this by using a significance level of α/m for each test individually. Example: Suppose the experimenter in the battery example collected the data, analyzed them, looked at the confidence intervals in the Minitab output, noticed that the estimate of the mean for the second level was largest and the estimate for the first level the second
2 largest, and tested the null hypothesis H0: µ1 = µ2. For what p-values should he reject the null hypothesis using the Bonferroni method in order to claim his result is significant at the .05 level? This brings up the concepts of pre-planned comparisons and data snooping. A pre-planned comparison is one identified before running the experiment. The experiment should be designed so that these are estimable and their variance is as small as possible. Data-snooping occurs when you look at your data after the experiment has been performed and decide something looks interesting, then do a test on it. There is nothing wrong with data-snooping -- often interesting results are found this way. But datasnooping tests need to be done with care to obtain an honest significance level. The problem is that they usually are the result of several comparisons, not just the one formally tested. So if, for example, a Bonferroni procedure is used, you need to take into account all the other comparisons that are done informally in setting a significance level. Summary of utility of Bonferroni methods: • Not recommended for data snooping -- it's too easy to overlook comparisons that were made in deciding what to test. • OK for pre-planned comparisons when m is small. • Not useful when m is large -- too conservative. 2. General Comments on Methods for Multiple Comparisons. There are many other methods for multiple comparison. All, like the Bonferroni method, produce confidence intervals with endpoints of the form Cˆ ± w se( Cˆ ),
where C is the contrast or other parameter being estimated, Cˆ is the least squares estimate of C, se( Cˆ ) is the standard error of Cˆ , and w (called the critical coefficient) depends on the overall confidence level α, on the method, on the number v of treatments, on the number m of things being estimated, and on the number of error degrees of freedom. For the Bonferroni method, w = wB = t(n-v, α/m) The half-width w se( Cˆ ) of the confidence interval is called the minimum significant difference (msd for short), because it is the smallest value of Cˆ that will produce a
3 confidence interval not containing 0, and hence say the contrast is significantly different from zero. 3. Scheffe Method. This method does not depend on the number of comparisons being made, but applies to contrasts only. The idea behind the method is that every contrast can be written as a linear combination of the v-1 "treatment vs control" contrasts τ2 - τ1, τ3 τ1, … , τv-1 - τ1. The method depends on finding a 1- α confidence region for these v-1 contrasts, and showing that this confidence region for these special contrasts determines confidence bounds for every possible contrast that are independent of the number of contrasts. Summary of utility of Scheffe method: • Does not matter how many comparisons are made, so suitable for data snooping. • If m is large, gives shorter confidence intervals than Bonferroni. • For m small, is "expensive insurrance." Since Minitab does not give the Scheffe method, we will not use it in this class. 4. Tukey Method for All Pairwise Comparisons. As the name suggests, this is used for all pairwise contrasts τi - τj. This is sometimes called the Honest Significant Difference Method, since (for equal sample sizes) it depends on the distribution of the statistic Q=
max{T1,..., Tv } − min{T1,..., Tv } , MSE r
where Ti = Yi⋅ - µi. This distribution is called the Studentized range distribution. It has two degrees of freedom. The critical coefficient is wT = q(v, n-v, α)/ 2 . For equal sample sizes, the overall confidence level is 1-α; for unequal sample sizes, it is at least 1-α. Note that since this method only deals with pairwise contrasts, the standard error 1 1 of τi - τj involved in the calculation of the msd is just msE + ri rj Summary of utility of Tukey method: • Gives shorter confidence intervals than either Bonferroni or Scheffe. • In basic form can be used only for pairwise comparisons. (There is an extension to all contrasts, but it is usually not as good as Scheffe.) Example: Battery Experiment.
4 5. Dunnett Method for Treatment-Versus-Control Comparisons. This is even more specialized than the Tukey method, but the special situation is often of interest. If we assume Treatment 1 is a control, then we are likely to be interested in the treatmentversus-control contrasts τi -τ1. This method is based on the joint distribution (a type of multivariate t-distribution) of the estimators Yi⋅ - Y1⋅ . Because the distribution is complicated, the calculation of wD is best left to reliable software. However, not all software (e.g., Minitab) gives one-sided confidence intervals, which might be desired. Summary of utility of Dunnett method: • Best method for treatment-versus-control comparisons. • Not applicable to other types of comparisons. Example: Battery experiment. 6. Hsu's Method for Multiple Comparisons with the Best Treatment. The idea is similar to Dunnett's method, but instead of comparing each treatment with the control group, each treatment is compared with the best of the other treatments. The procedure varies slightly depending on whether "best" is largest or smallest. Minitab allows the user to check which is desired. Summary of utility of Hsu method: • Good for what it does. • Not applicable to other types of comparisons. Example: Battery experiment. 7. Other Methods. There are many. Books have been written on the subject (e.g., Miller, Hsu). Some people have their favorites, which others argue are not good choices.