Hands-On Unsupervised Learning Using Python

Chapter 11. Feature Detection Using Deep Belief Networks

In Chapter 10, we explored restricted Boltzmann machines and used them to build a recommender system for movie ratings. In this chapter, we will stack RBMs together to build deep belief networks (DBNs). DBNs were first introduced by Geoff Hinton at the University of Toronto in 2006.

RBMs have just two layers, a visible layer and a hidden layer; in other words, RBMs are just shallow neural networks. DBNs are made up of multiple RBMs—the hidden layer of one RBM serves as the visible layer of the next RBM. Because they involve many layers, DBNs are deep neural networks. In fact, they are the first type of deep unsupervised neural network we’ve introduced so far.

Shallow unsupervised neural networks, such as RBMs, cannot capture structure in complex data such as images, sound, and text, but DBNs can. DBNs have been used to recognize and cluster images, video capture, sound, and text, although other deep learning methods have surpassed DBNs in performance over the past decade.

Deep Belief Networks in Detail

Like RBMs, DBNs can learn the underlying structure of input and probabilistically reconstruct it. In other words, DBNs—like RBMs—are generative models. And, as with RBMs, the layers in DBNs have connections only between layers but not between units within each layer.

In the DBN, one layer is trained at a time, starting with the very first hidden layer, which, along with the input layer, makes up the first RBM. Once this first ...