Programming Machine Learning

What You Just Learned

Deep neural networks can be wild, untameable beasts. In this chapter, you learned of a few useful techniques to turn them into cute docile puppies.

We started with a lengthy discussion of activation functions. You learned that nonlinear functions are a necessity in neural networks, but we must choose them carefully. So far we used sigmoids inside the network, but sigmoids can cause a number of problems as the network gets deeper: saddening dead neurons, perplexing vanishing gradients, and shocking exploding gradients. For that reason, we looked at a few alternatives to the sigmoid—in particular, the popular ReLU activation function.

After that discussion of activation functions, we took a whirlwind tour through a number ...