Machine Learning Algorithms - Second Edition

Batch normalization layers

When the network is very deep, it's possible to observe a progressive modification in the mean and standard deviation of the batches throughout the network. This phenomenon is called covariate shift and is mainly responsible for a performance loss in terms of training speed. A batch normalization layer is responsible for correcting the statistical parameters of each batch and is normally inserted after a standard layer. Analogously to dropout, batch normalization operates only during the training phase, but, in this case, the model will apply a normalization computed over all samples during the prediction phase. It has been observed that, as a secondary effect, the batch normalization layers provide a regularization ...