Hands-On Artificial Intelligence for Beginners

Modern approaches to activation functions

In recent years, several modifications have been made to the ReLU function to solve the dying neuron problem and make them more robust. Most notable of these is a solution called the Leaky ReLU. Leaky ReLU introduces a small slope to the ReLU function to keep potentially dead neurons alive, by allowing a small gradient to keep the units active. The Leaky ReLu function is available in TensorFlow as:

tf.nn.leaky_relu(features,alpha=0.2,name=None)

Another adaption, the Parametric Rectified Linear Unit (PreLU), takes this further by making that small gradient a parameter that can be adjusted during training. Instead of predefining a slope of the function, the slope becomes an adaptable parameter, hence ...