Machine Learning, 2nd Edition

Chapter 8Consistency

When you have data, you analyze the data. When you don’t have data, you prove theorems.

By this point, the mathematically sophisticated reader may, upon consideration of all of the classification and regression methods presented in Chapters 3, 4, and 6, throw up their hands in frustration due to the feeling that such a wide array of different methods is a sign not of knowledge and understanding, but of ignorance. “Surely,” such a reader might say, “there ought to be one method that solves all classification problems optimally, at least as the number of data, , increases without bound?”

This short chapter answers that question in the affirmative, introducing the reader to a property of classification methods known as consistency. A classification method is consistent for a given classification problem if the risk of the trained classifier(s) it produces converges to the risk of the Bayes classifier for that problem as (for some appropriate definition of convergence). A classification method is universally consistent if it is consistent with respect to all joint distributions on a given domain and range, .

The main results presented in this chapter are: there ...