Hands-On Deep Learning Algorithms with Python

The leaky ReLU function

Leaky ReLU is a variant of the ReLU function that solves the dying ReLU problem. Instead of converting every negative input to zero, it has a small slope for a negative value as shown:

Leaky ReLU can be expressed as follows:

The value of is typically set to 0.01. The leaky ReLU function is implemented as follows:

def leakyReLU(x,alpha=0.01):    if x<0:        return (alpha*x)    else:        return x

Instead of setting some default values ...