40 Algorithms Every Programmer Should Know

The loss and cost functions

The loss function defines how we want to quantify an error for a particular example in our training data. The cost function defines how we want to minimize an error in our entire training dataset. So, the loss function is used for one of the examples in the training dataset and the cost function is used for the overall cost that quantifies the overall deviation of the actual and predicted values. It is dependent on the choice of w and h.

The loss function used in logistic regression is as follows:

Loss (ý⁽ⁱ⁾, y⁽ⁱ⁾) = - (y⁽ⁱ⁾ log ý⁽ⁱ⁾ + (1-y⁽ⁱ⁾ ) log (1-ý⁽ⁱ⁾)

Note that when y⁽ⁱ⁾ = 1, Loss(ý⁽ⁱ⁾, y⁽ⁱ⁾) = - logý⁽ⁱ⁾. Minimizing the loss will result in a large value of ý⁽ⁱ⁾ . Being a sigmoid function, the maximum ...