Deep Learning Essentials

Regularization

One of the challenges in training CNNs is overfitting. Overfitting can be defined as a phenomenon where CNN, or in general any learning algorithm, performs very well in optimizing training error, but is not able to generalize well on test data. The most common trick used in the community to address this issue is regularization, which is simply adding a penalty to the loss function being optimized. There are various ways of regularizing the network. Some of the common techniques are explained as follows:

• L2 regularization: One the most popular forms of regularization, an L2 regularizer implements a squared penalty on the weights, meaning, the higher the weights, the higher the penalty. This ensures once the network is trained, ...