Hands-On Machine Learning for Algorithmic Trading

Hidden units

Hidden units are unique to the design of neural networks, and several non-linear activation functions have been used successfully. The design of hidden activation functions remains an area of research because it has a critical impact on the training process.

A very popular class of activation functions are piece-wise linear units, such as the Rectified Linear Unit (ReLU). The functional form is similar to the payoff for a call option and the activation is computed as g(z) = max(0, z) for a given activation, z. As a result, the derivative is constant whenever the unit is active. ReLUs are usually combined with an affine transformation of the inputs. They are often used instead of sigmoid units and their discovery has greatly ...