Hands-On Convolutional Neural Networks with TensorFlow

Dropout

Another technique for regularization that we will look at is something called Dropout. Introduced in 2012 by G. E. Hinton, dropout is a simple method of regularization that gives very good results. The idea behind dropout is that at each training iteration, all the neurons in a layer may, with random probability (usually 50%), be turned on and off.

This turning on and off forces the network to learn the same concepts as usual, but via multiple different paths. After training, all neurons are kept on, and these paths will behave like an ensemble of multiple networks that will be used to average the final result, thus improving generalization. It forces the weight to be distributed across the whole network and keeps the low somewhat ...