Hands-On Deep Learning for IoT

Weight and bias initialization

Now, here's a tricky question: how do we initialize the weights? Well, if we initialize all the weights to the same value (for example, 0 or 1), each hidden neuron will get the same signal. Let's try to break it down:

• If all weights are initialized to 1, then each unit gets a signal equal to the sum of the inputs.
• If all weights are 0, which is even worse, then every neuron in a hidden layer will get zero signal.

For network weight initialization, Xavier initialization is used widely. It is similar to random initialization, but often turns out to work much better, since it can identify the rate of initialization depending on the total number of input and output neurons by default. You may be wondering whether ...