Hands-On Mathematics for Deep Learning

Rectified linear unit

Rectified linear unit (ReLU) is one of the most widely used activation functions because it is more computationally efficient than the activation functions we have already seen; therefore, it allows the network to train a lot faster and so converge more quickly.

The ReLU function is as follows:

The function looks as follows:

As you can see, all the negative values for x are clipped off and turn into 0. It may surprise you to know that even though this looks like a linear function, it has a derivative that is as follows: ...