Mastering Machine Learning Algorithms

Example of batch normalization with Keras

In order to show the feature of this technique, let's repeat the previous example using an MLP without dropout but applying a batch normalization after each fully connected layer before the ReLU activation. The example is very similar to the first one, but, in this case, we increase the Adam learning rate to 0.001 keeping the same decay:

from keras.models import Sequentialfrom keras.layers import Dense, Activation, BatchNormalizationfrom keras.optimizers import Adammodel = Sequential()model.add(Dense(2048, input_shape=(width * height, )))model.add(BatchNormalization())model.add(Activation('relu'))model.add(Dense(1024))model.add(BatchNormalization())model.add(Activation('relu'))model.add(Dense(1024)) ...