Programming Collective Intelligence

Chapter 10. Finding Independent Features

Most of the chapters so far have focused primarily on supervised classifiers, except Chapter 3, which was about unsupervised techniques called clustering. This chapter will look at ways to extract the important underlying features from sets of data that are not labeled with specific outcomes. Like clustering, these methods do not seek to make predictions as much as they try to characterize the data and tell you interesting things about it.

You’ll recall from Chapter 3 that clustering assigns every row in a dataset to a group or point in a hierarchy—each item fits into exactly one group that represents the average of the members. Feature extraction is a more general form of this idea; it tries to find new data rows that can be used in combination to reconstruct rows of the original dataset. Rather than belonging to one cluster, each row is created from a combination of the features.

One of the classic problems illustrating the need to find independent features is known as the cocktail party problem, the problem of interpreting conversation when many people are talking. A remarkable feature of the human auditory system is our ability to focus on a single voice in a room full of people talking, despite the fact that a mixture of all the different voices is reaching our ears. The brain is quite adept at separating the independent sounds that create all the noise it’s hearing. By using algorithms like the one described in this chapter, and getting ...