Hands-On Deep Learning with R

Plotting the rectified linear units activation function

Rectified Linear Units (ReLU) is a hybrid function that fits a line for positive values of x while assigning any negative values of x with a value of 0. Even though one half of this function is linear, the shape is nonlinear and carries with it all the advantages of nonlinearity, such as being able to use the derivative for backpropagation.

Unlike the previous two activation functions, it has no upper bound. This lack of a constraint can be helpful to avoid the issue with the sigmoid or tanh function, where the gradient becomes very gradual near the extremes and provides little information to help the model continue to learn. Another major advantage of ReLU is how it leads to sparsity ...