Python Deep Learning - Second Edition

Vanishing and exploding gradients

The preceding example has an issue, though. Let's run the training with a longer sequence:

x = np.array([[0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0]])y = np.array([12])losses, weights_u, weights_w = train(x, y, epochs=150)plot_training(losses, weights_u, weights_w)

The output is:

chapter_07_001.py:5: RuntimeWarning: overflow encountered in multiply     return x * U + s * Wchapter_07_001.py:40: RuntimeWarning: invalid value encountered in multiply     gU += np.sum(gS * x[:, k - 1])chapter_07_001.py:41: RuntimeWarning: invalid value encountered in multiply     gW += np.sum(gS * s[:, k - 1])

The reason for these warnings is that the final parameters ...