Machine Learning Algorithms - Second Edition

Dropout layers

A dropout layer is used to prevent the overfitting of the network by randomly setting a fixed number of input elements to zero (generally, this is achieved by setting the probability of elements to nullify). This layer is adopted during the training phase, but it's normally deactivated during the test, validation, and production phases. Dropout networks can exploit higher learning rates, moving in different directions on the loss surface (setting to zero a few random input values in the hidden layers is equivalent to training different sub-models) and excluding all the error-surface areas that don't lead to a consistent optimization. Dropout is very useful in very big models, where it increases the overall performance and reduces ...