April 2018
Intermediate to advanced
334 pages
10h 18m
English
The rectified linear unit, better known as ReLU, is the most widely used activation function:

The ReLU function has the advantage of being non linear. Thus, backpropagation is easy and can therefore stack multiple hidden layers activated by the ReLU function, where for x<=0, the function f(x) = 0 and for x>0, f(x)=x.

The main advantage of the ReLU function over other activation functions is that it does not activate all the neurons at the same time. This can be observed from the preceding graph of the ReLU ...