Theory of Ridge Regression Estimation with Applications

5Multiple Linear Regression Models

5.1 Introduction

Traditionally, we use least squares estimators (LSEs) for a linear model which provide minimum variance unbiased estimators. However, data analysts point out two deficiencies of LSEs, namely, the prediction accuracy and the interpretation. To overcome these concerns, Tibshirani (1996) proposed the least absolute shrinkage and selection operator (LASSO). It defines a continuous shrinking operation that can produce coefficients that are exactly zero and is competitive with subset selection and ridge regression estimators (RREs), retaining the good properties of both the estimators. The LASSO simultaneously estimates and selects the coefficients of a given linear model.

However, the preliminary test estimator (PTE) and the Stein‐type estimator only shrink toward the target value and do not select coefficients for appropriate prediction and interpretation.

LASSO is related to the estimators, such as nonnegative garrote by Breiman (1996), smoothly clipped absolute derivation (SCAD) by Fan and Li (2001), elastic net by Zou and Hastie (2005), adaptive LASSO by Zou (2006), hard threshold LASSO by Belloni and Chernozhukov (2013), and many other versions. A general form of an extension of LASSO‐type estimation called the bridge estimation, by Frank and Friedman (1993), is worth pursuing.

This chapter is devoted to the comparative study of the finite sample performance of the primary penalty estimators, namely, LASSO and the RREs. They ...