Artificial Intelligence By Example

Binary cross-entropy

Cross-entropy comes in handy when the learning slows down. In the hill example, it slowed down at the bottom. But, remember, a path can lead you sideways, meaning you are momentarily stuck at a given height. Cross-entropy solves that by being able to function well with very small values (steps on the hill).

Suppose you have the following structure:

• Inputs is {x1, x2, ..., xn}
• Weights is {w1, w2, ..., wn}
• A bias (or sometimes more) is b
• An activation function(ReLU, logistic sigmoid, or other)

Before the activation, z represents the sum of the classical operations:.

Now the activation function is applied to z to obtain ...