Deep Learning By Example

Avoiding the curse of dimensionality

In the previous sections, we showed that the classifier's performance will decrease when the number of features exceeds a certain optimal point. In theory, if you have infinite training samples, the curse of dimensionality won't exist. So, the optimal number of features is totally dependent on the size of your data.

An approach that will help you to avoid the harm of this curse is to subset M features from the large number of features N, where M << N. Each feature from M can be a combination of some features in N. There are some algorithms that can do this for you. These algorithms somehow try to find useful, uncorrelated, and linear combinations of the original N features. A commonly used technique for ...