Deep Learning Essentials

Regularization

Like all other machine learning approaches, overfitting is something that needs to be controlled all the time, especially given that networks have so many parameters to learn. One of the methods to deal with overfitting is called regularization. Typical regularization is done by adding some constraints on the parameters, such as L1 or L2 regularization, which prevent the weights or coefficients of the networks growing too big. Take L2 regularization as an example. It is achieved by augmenting the cost function with the squared magnitude of all weights in the neural network. What it does is to heavily penalize the peaky weight vectors and diffuse the weight vectors.

That is, we encourage the network to use all of its input rather ...