Deep Learning Essentials

Xavier initialization

Xavier Glorot and Yoshua Bengio proposed another way of initialization called Xavier in their paper, Understanding the difficulty of training deep feedforward neural networks (http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf). Their major goal was to prevent gradient vanishing and too-large weight problems (as backpropagation grows proportionally to the value of the weights). In other words, the Xavier initialization tries to solve the following two problems at the same time:

• If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful
• If the weights in a network start too large, then the signal grows as it passes through each layer until ...