Common Errors in Statistics (and How to Avoid Them), 4th Edition

BOOTSTRAP

Many of the procedures discussed in this chapter fall victim to the erroneous perception that one can get more out of a sample or series of samples than one actually puts in. One bootstrap expert learned he was being considered for a position because management felt, “your knowledge of the bootstrap will help us to reduce the cost of sampling.”

Michael Chernick, author of Bootstrap Methods: A Practitioner’s Guide [2007], has documented six myths concerning the bootstrap:

1. Allows you to reduce your sample size requirements by replacing real data with simulated data—Not. Kwon and Moon [2006] made precisely this error in applying the bootstrap to assess the probability of dam overflow.

2. Allows you to stop thinking about your problem, the statistical design and probability model—Not.

3. No assumptions necessary—Not. One particular but remediable assumption is that the observations be independent. In the case of time series, where adjacent observations may be dependent, the use of moving-block [Künsch, 1989] or circular block [Politis and Romano, 1992] bootstraps is recommended.

4. Can be applied to any problem—Not.

5. Only works asymptotically—Necessary sample size depends on the context.

6. Yields exact significance levels—Never.

To which we would add never use the bootstrap (or any other method) to test a hypothesis if a more powerful method is available. For example, Derado et al. [2004] performed a series of complex time-consuming measurements on 12 difficult ...