Programming Machine Learning

Chapter 18Taming Deep Networks

In a sense, there’s nothing special about deep networks. They’re like shallow neural networks, only with more layers. When people started experimenting with deep networks, however, they faced an uncomfortable truth: building deep networks may be easy, but training them is not.

Backpropagation on deep networks comes with its own specific challenges that carry intimidating names such as “vanishing gradients” and “dead neurons.” Those challenges rarely come up in shallow neural networks—but they’re par for the course in deep neural networks.

Over the years, neural networks researchers developed a collection of strategies, or what you might call a “bag of tricks,” to tackle those challenges and tame deep neural networks: ...