Math for Deep Learning

10BACKPROPAGATION

Backpropagation is currently the core algorithm behind deep learning. Without it, we cannot train deep neural networks in a reasonable amount of time, if at all. Therefore, practitioners of deep learning need to understand what backpropagation is, what it brings to the training process, and how to implement it, at least for simple networks. For the purposes of this chapter, I’ll assume you have no knowledge of backpropagation.

We’ll begin the chapter by discussing what backpropagation is and what it isn’t. We’ll then work through the math for a trivial network. After that, we’ll introduce a matrix description of backpropagation ...