Programming Machine Learning

The Case for Backpropagation

Soon enough, we’re going to explain backpropagation from the ground up. But first, let’s see why people use backpropagation in the first place.

To begin with, here’s a piece of good news: in a sense, you already know how to train a neural network. You train it with gradient descent, like you train a perceptron. At each iteration, you calculate the gradient of the loss, then descend that gradient to minimize the loss. (If you need a refresher, review Chapter 3, ​Walking the Gradient​.)

Now for the less-than-good news: descending the gradient is the easy part of the job. The tough part is calculating that gradient in the first place.

In the case of the perceptron, we knew how to get the gradient: we calculated the ...