Getting started with deep learning

Chapter 2. Fundamentals of Deep Networks

Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!

The Red Queen, Through the Looking Glass

Defining Deep Learning

In the Chapter 1 we set up the foundations of machine learning and neural networks. In this chapter we’ll build on these foundations to give you the core concepts of deep networks. This will help build your understanding of what is going on in different network architectures as we progress into the specific architectures in Chapter 3. Let’s begin by restating our definitions of both deep learning and deep networks.

What Is Deep Learning?

The facets that differentiate deep learning networks in general from “canonical” feed-forward multilayer networks are as follows:

• More neurons than previous networks
• More complex ways of connecting layers
• “Cambrian explosion” of computing power to train
• Automatic feature extraction

When we say “more neurons,” we mean that the neuron count has risen over the years to express more complex models. Layers also have evolved from each layer being fully connected in multilayer networks to locally connected patches of neurons between layers in Convolutional Neural Networks (CNNs) and recurrent connections to the same neuron in Recurrent Neural Networks (in addition to the connections from the previous layer).

More connections means that our networks have more parameters to optimize, ...