Hands-On Deep Learning with Go

Rectified Linear Units

Rectified Linear Unit (ReLU) is the most popular activation function in use. We will be using it as the primary activation function in a number of advanced architectures in later chapters.

It can be described as follows:

func relu(x){   return Max(0,x)}

If we were to chart it out, it looks something like the following diagram:

As you can see, it is extremely similar to a linear function, except that it goes to zero (therefore indicating that the neuron is not activated).

ReLU also has many useful properties, as follows:

• It is nonlinear: Therefore, stacking several layers of these will not necessarily result in being ...