R Deep Learning Essentials - Second Edition

L1 penalty in action

To see how the L1 penalty works, we can use a simulated linear regression problem. The code for the rest of this chapter is in Chapter3/overfitting.R. We simulate the data, using a correlated set of predictors:

set.seed(1234)X <- mvrnorm(n = 200, mu = c(0, 0, 0, 0, 0),  Sigma = matrix(c(    1, .9999, .99, .99, .10,    .9999, 1, .99, .99, .10,    .99, .99, 1, .99, .10,    .99, .99, .99, 1, .10,    .10, .10, .10, .10, 1  ), ncol = 5))y <- rnorm(200, 3 + X %*% matrix(c(1, 1, 1, 1, 0)), .5)

Next, we can fit an OLS regression model to the first 100 cases, and then use lasso. To use lasso, we use the glmnet() function from the glmnet package. This function can actually fit the L1 or the L2 (discussed in the next section) penalties, and which ...