Deep Learning from Scratch

Chapter 4. Extensions

In the last chapter, after having spent two chapters reasoning from first principles about what deep learning models are and how they should work, we finally built our first deep learning model and trained it to solve the relatively simple problem of predicting house prices given numeric features about houses. On most real-world problems, however, successfully training deep learning models isn’t so easy: while these models can conceivably find an optimal solution to any problem that can be framed as a supervised learning problem, in practice they often fail, and indeed there are few theoretical guarantees that a given model architecture will in fact find a good solution to a given problem. Still, there are some well-understood techniques that make neural network training more likely to succeed; these will be the focus of this chapter.

We’ll start out by reviewing what neural networks are “trying to do” mathematically: find the minimum of a function. Then I’ll show a series of techniques that can help the networks achieve this, demonstrating their effectiveness on the classic MNIST dataset of handwritten digits. We’ll start with a loss function that is used throughout classification problems in deep learning, showing that it significantly accelerates learning (we’ve only covered regression problems thus far in this book because we hadn’t yet introduced this loss function and thus haven’t been able to do classification problems justice). On a similar note, ...