Hands-On Deep Learning Algorithms with Python

The Swish function

The Swish function is a recently introduced activation function by Google. Unlike other activation functions, which are monotonic, Swish is a non-monotonic function, which means it is neither always non-increasing nor non-decreasing. It provides better performance than ReLU. It is simple and can be expressed as follows:

Here, is the sigmoid function. The Swish function is shown in the following diagram:

We can also reparametrize ...