Hands-On Machine Learning for Algorithmic Trading

The detector stage – adding non-linearity

The feature maps are usually passed through a non-linear transformation. The rectified linear unit (ReLU) that we encountered in the last chapter is a common function for this purpose. ReLUs replace negative feature map activations element-wise by zero.

A popular alternative is the Softplus function:

In contrast to ReLU, it has a derivative everywhere, which is the sigmoid function that we used for logistic regression (see Chapter 7, Linear Models).