Programming Machine Learning

Enter the Softmax

Check out the activation functions in our neural network. So far, we took it for granted that both of those functions are sigmoids. However, most neural networks replace the last sigmoid, the one right before the output layer, with another function called the softmax.

Let me show you what the softmax looks like, and then we’ll see why it’s useful. Like the sigmoid, the softmax takes an array of numbers, that in this case are called the logits, and returns an array with the same size as the input. Here is the formula of the softmax, in case you want to understand the math behind it:

You can read this formula as: take the exponential ...