Programming Machine Learning

What You Just Learned

In this chapter, you learned backpropagation—an algorithm to calculate the gradients of the weights in a neural network. Those gradients represent the impact of each weight on the overall loss.

Each training iteration in a neural network ping-pongs between two steps:

1. Forward propagation: the network calculates each layer from the previous one, from the input layer to the output ŷ.

2. Backpropagation: the network bounces its way back from the output layer to the weights, using the chain rule to calculate their gradients. Then it descends those gradients, pushing the loss down, and ŷ closer to the ground truth y.

You also learned that neural networks don’t train well if all the weights have the same value. Instead, you should ...