Deep Learning with TensorFlow 2 and Keras - Second Edition

15

The Math Behind Deep Learning

In this chapter we discuss the math behind deep learning. This topic is quite advanced and not necessarily required for practitioners. However, it is recommended reading if you are interested in understanding what is going on under the hood when you play with neural networks. We start with an historical introduction, and then we will review the high school concept of derivatives and gradients. We will also introduce the gradient descent and backpropagation algorithms commonly used to optimize deep learning networks.

History

The basics of continuous backpropagation were proposed by Henry J. Kelley [1] in 1960 using dynamic programming. Stuart Dreyfus proposed using the chain rule in 1962 [2]. Paul Werbos was ...