Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!
The Red Queen, Through the Looking Glass
In the Chapter 1 we set up the foundations of machine learning and neural networks. In this chapter we’ll build on these foundations to give you the core concepts of deep networks. This will help build your understanding of what is going on in different network architectures as we progress into the specific architectures in Chapter 3. Let’s begin by restating our definitions of both deep learning and deep networks.
The facets that differentiate deep learning networks in general from “canonical” feed-forward multilayer networks are as follows:
When we say “more neurons,” we mean that the neuron count has risen over the years to express more complex models. Layers also have evolved from each layer being fully connected in multilayer networks to locally connected patches of neurons between layers in Convolutional Neural Networks (CNNs) and recurrent connections to the same neuron in Recurrent Neural Networks (in addition to the connections from the previous layer).
More connections means that our networks have more parameters to optimize, ...