Conformal Prediction for Reliable Machine Learning

Chapter 7

Model Selection

David R. Hardoon^a, Zakria Hussain^b and John Shawe-Taylor^b,    ^aErnst & Young, Singapore,    ^bDepartment of Computer Science, University College London, United Kingdom

Abstract

We investigate the issue of model selection and the use of the nonconformity (strangeness) measure in batch learning. Using the nonconformity measure we propose a new training algorithm that helps avoid the need for Cross-Validation or Leave-One-Out model selection strategies. We provide a new generalization error bound using the notion of nonconformity to upper bound the loss of each test example and show that our proposed approach is comparable to standard model selection methods, but with theoretical guarantees of success and faster convergence. ...