Hands-On Deep Learning with R

Defining the swish activation function

Swish is a more recently developed activation function that aims to leverage the strengths of ReLU while also addressing some of its shortcomings. Swish, like ReLU, has a lower bound and no upper bound, which is a strength as it can still deactivate neurons while preventing values from being forced to converge around an upper bound. However, unlike ReLU, the lower bound is still curved, and what's more notable is that the line is nonmonotonic, which means that as values for x decrease, the value for y can increase. This is an important feature that prevents the dying neuron problem as the derivative can continue to be modified across iterations.

Let's investigate the shape of this activation function: ...