Deep Learning Essentials

Loss functions

So far, we have seen a CNN is trained using a gradient descent-based algorithm that tries to minimize a loss function given the training data. There are multiple ways to define how to choose this loss function. In this section, we will look at some of the most commonly used loss functions used for CNN training:

• Cross-entropy loss: This is one of the most popularly used loss functions for CNNs. It is based on the notion of cross-entropy, which is a measure of distance between a true distribution and an estimated distribution , and can be defined as . Using this measure, the cross-entropy loss can be defined as follows: