R Deep Learning Essentials - Second Edition

Grid search

For more information on tuning hyper-parameters, see Bengio, Y. (2012), particularly Section 3, Hyperparameters, which discusses the selection and characteristics of various hyper-parameters. Aside from manual trial and error, two other approaches for improving hyper-parameters are grid searches and random searches. In a grid search, several values for hyper-parameters are specified and all possible combinations are tried. This is perhaps easiest to see. In R, we can use the expand.grid() function to create all possible combinations of variables:

expand.grid( layers=c(1,4), lr=c(0.01,0.1,0.5,1.0), l1=c(0.1,0.5))   layers    lr   l11       1  0.01  0.12       4  0.01  0.13       1  0.10  0.14       4  0.10  0.15       1  0.50  0.16       4  0.50  0.17       1  1.00  0.18       4  1.00  0.19 1 0.01 ...