Machine Learning with Swift

Step-like activation functions

The heaviside step function (also known as unit step function or threshold function) outputs 0 for all values less than zero, and 1 for everything else. This is a natural choice to model a biological neuron, which produces an electrical impulse, 1, or stays silent: 0. Unfortunately, the function is not differentiable because of the discontinuity at 0, which makes it impossible to train such networks using gradient descent algorithm. Each individual neuron in such a network is a mathematical equivalent of a binary linear classifier, hence such networks are unable to perform well on nonlinear tasks.

A logistic (sigmoid) function is a continuous approximation of a step function. The function squashes the input ...