Methods and Applications of Statistics in Clinical Trials, Volume 2: Planning, Analysis, and Inferential Methods

Chapter 25

Multiple Comparisons

Michael A. Proschan

25.1 Introduction

Multiple comparisons arise in several ways in clinical trials; common examples include multiple arms, multiple end points, subgroup analyses, and monitoring over multiple time points. At first, it seems that trying to answer several questions in the same trial is very efficient, especially if another large trial to answer the same questions is unlikely. The problem is that if the comparisonwise error rate (also known as the per-comparison error rate)—the expected proportion of false positives—is 0.05, then the chance of at least one false positive, which is known as the familywise error rate (FWE) or the experimentwise error rate, can be substantially greater than 0.05. In fact, with enough comparisons, the FWE can be close to 1. To control the FWE at level 0.05, one must require stronger evidence for each individual comparison; this is called a multiple comparison adjustment.

25.2 Strong and Weak Control of the FWE

Two different levels of control of the FWE exist. The first level, which is called weak control, means that under the global null hypothesis that each separate null hypothesis is true, the probability of at least one type 1 error is no greater than α. But the global null hypothesis is a very strong assumption; it is more likely that some null hypotheses are true and others are not. We would like the probability of rejecting at least one true null hypothesis to be α or less, where that probability ...