Deep Learning with PyTorch Quick Start Guide

Batch normalization

Batch normalization is used widely to improve the performance of neural networks. It works by stabilizing the distributions of layer input. This is achieved by adjusting the mean and variance of these input. It is fairly indicative of the nature of deep learning research that there is uncertainty among the researcher community as to why batch normalization is so effective. It was thought that this was because it reduces the so called internal co-variate shift (ICS). This refers to the change in distributions as a result of the preceding layers' parameter updates. The original motivation for batch normalization was to reduce this shift. However, a clear link between ICS and performance has not been conclusively found. More ...