Hands-On Transfer Learning with Python

Initialization of model parameters

Here's how the choice of initial points impacts the performance of the iterative learning algorithm for deep neural networks:

• Initial points can determine whether the learning will converge or not
• Even if learning converges, how quickly it converges depends on the initial point
• Initial points of comparable costs may have varying generalization errors

Initialization algorithms are mostly heuristics. The whole point of a good initialization is having one that somehow enables faster learning. One of the important aspects of initialization is to break symmetry of the initial set of weights to the units of hidden layers. Two units with same activation function at the same level of the network will be updated ...