Programming Machine Learning

From the Chain Rule to Backpropagation

Backpropagation is an application of the chain rule, one of the fundamental rules of calculus. Let’s see how the chain rule works on a couple of network-like structures: a simpler one, and a more complicated one.

The Chain Rule on a Simple Network

Look at this simple network-like structure:

This one isn’t a neural network, because it doesn’t have weights. Let’s borrow a term from computer science, and call it a computational graph. This graph has an input a, followed by two operations: “multiply by two” and “square.” The output of the multiplication is called b, and the output of the entire graph is called ...