Python Deep Learning - Second Edition

Batch normalization

In Data pre-processing, we explained why data normalization is important. Batch normalization provides a way to apply data processing, similar to the standard score, for the hidden layers of the network. It normalizes the outputs of the hidden layer for each mini-batch (hence the name) in a way, which maintains its mean activation value close to 0, and its standard deviation close to 1. We can use it with both convolutional and fully-connected layers. Networks with batch normalization train faster and can use higher learning rates. For more information about batch normalization, see the original paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, by Sergey Ioffe and Christian ...