O'Reilly logo

Apache Spark 2.x Machine Learning Cookbook by Shuen Mei, Broderick Hall, Meenakshi Rajendran, Siamak Amirghodsi

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Introduction

The curse of dimensionality is not a new term or concept. The term was originally coined by R. Bellman when tackling problems in dynamic programming (the Bellman equation). The core concepts in machine learning refer to the problem that as we increase the number of dimensions (axes or features), the number of training data (samples) remains the same (or relatively low), which causes less accuracy in our predictions. This phenomenon is also referred to as the Hughes Effect, named after G. Hughes, which talks about the problem caused by rapid (exponential) increase of search space as we introduce more and more dimensions to the problem space. It is a bit counterintuitive, but if the number of samples does not expand at the same ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required