Machine Learning for Developers

Rectified linear unit or ReLU

ReLU is called a rectified linear unit, and one of its main advantages is that it is not affected by vanishing gradient problems, which generally consist of the first layers of a network tending to be values of zero, or a tiny epsilon:

    class Relu(TransferFunction):
        def getTransferFunction(x):
            return x * (x>0)
        def getTransferFunctionDerivative(x):
            return 1 * (x>0)
    graphTransferFunction(Relu)

Let's take a look at the following graph: