Hands-On Deep Learning with Go

Linear functions

A possible extension to the step function might be to use a linear function, as shown in the following code:

func linear(x){   return 0.5 * x}

This is still very simple and, if we were to chart it out, it would look something like the following diagram:

However, this function is still not very useful. If we were to look at the gradient, we'll see that, when we differentiate this function, all we get is a straight line equal to the value of a. This means it suffers the same problem as the step function; that is to say, we won't see much improvement from backpropagation.

In addition, if we were to stack several layers of this, ...