Hands-On Transfer Learning with Python

Local minima and saddle points

DNN models are essentially guaranteed to have an extremely large number of local minima. If a local minimum has a high cost compared to a global minima, this could be problematic. For a long time, it was assumed that, due to the existence of such local minima, neural network training is plagued. This is still an active research area, but now it's suspected that for DNNs, most of the local minima have low cost value and it's not necessary to a find global minimum, but rather a point in the weight space where we have a sufficiently low value of the cost function. The existence of strong local minima can be detected by monitoring the gradient norm. If the gradient norm decreased to an insignificantly small order, ...