Hands-On Artificial Intelligence for Beginners

Activation functions

Activation functions are the building blocks that make neural networks able to do what they do: convert inputs into desired outputs within ANNs in a nonlinear fashion. As such, they are frequently referred to as nonlinearities. Putting this together with what we learned earlier, in a neural network we compute the sum of products of input (X) and their corresponding weights w, and apply an activation function f (x) to it to get the output of that layer and feed it as an input to the next layer.

Without a nonlinearity, a unit would be just a simple linear function, and our network something such as a linear regression. When we think about traditional analytical models, such as linear regression or support vector machines, ...