Hands-On Convolutional Neural Networks with TensorFlow

Activation functions

In order to allow the ANN models to be able to tackle more complex problems, we need to add a nonlinear block just after the neuron dot product. If we then cascade these nonlinear layers, it allows the network to compose different concepts together, making complex problems easier to solve.

The use of nonlinear activations in our neurons is very important. If we didn't use nonlinear activation functions, then no matter how many layers we cascaded we would only ever have something that behaves like a linear model. This is because any linear combination of linear functions collapses down to be a linear function.

There are a wide variety of different activation functions that can be used in our neurons, and some are shown ...