R Deep Learning Essentials - Second Edition

Weight decay (L2 penalty in neural networks)

We have already unknowingly used regularization in the previous chapter. The neural network we trained using the caret and nnet package used a weight decay of 0.10. We can investigate the use of weight decay by varying it, and tuning it using cross-validation:

1. Load the data as before. Then we create a local cluster to run the cross-validation in parallel:

set.seed(1234)## same data as from previous chapterif (!file.exists('../data/train.csv')){  link <- 'https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/data/mnist_csv.zip'  if (!file.exists(paste(dataDirectory,'/mnist_csv.zip',sep="")))    download.file(link, destfile = paste(dataDirectory,'/mnist_csv.zip',sep="")) unzip(paste(dataDirectory,'/mnist_csv.zip',sep=""), ...