Adaptive Tests of Significance Using Permutations of Residuals with R and SAS

CHAPTER 4

PERMUTATION TESTS WITH LINEAR MODELS

4.1 INTRODUCTION

In Chapter 3 we discussed permutation methods for two-sample tests and we permuted the residuals to perform that test. We also showed that permutations of residuals was equivalent to permutations of observations in that two-sample test. Both of the methods are valid, so there is little controversy about the permutation method.

In contrast, when different permutation methods are used to test a coefficient in a linear regression model the results are not equivalent and it is not clear which method is superior. Some of the methods that have been proposed are:

• Permute the residuals from the reduced model and add these to the predicted values from the reduced model to form a new vector of predicted values.
• Permute the independent variable for the coefficient that is to be tested.
• Permute the dependent variable.
• Permute the residuals from the complete model and add these to the predicted values from the complete model to form a new vector of predicted values.

We will describe the first three of these methods in some detail in this chapter. For details concerning the last method see Anderson and Legendre (1999).

Usually we are not able to tabulate the full permutation distribution, so we take a sample of permutations from a population of all possible permutations. We could use sampling with replacement or sampling without replacement. Sampling without replacement could slightly reduce the variability in the p-value, but ...