The Big R-Book

♣25♣Model Validation

The modeller has a vast toolbox of measures to check the power of the model: lift, MSE, AUC, Gini, KS and many more (for more inspiration, see Chapter 25.1 “Model Quality Measures” on page 476). However, the intrinsic quality of the model is not the only thing to worry about. Even if it fits really good the data that we have, how will it behave on new data?

First, we need to follow up the power of the model. For example, we choose KS as the key parameter together with a confusion matrix. This means that as new data comes in we build a dashboard (for example see Chapter 36.3 “Dashboards” on page 725) that will read in new data and new observations of good and bad outcomes, and as new data becomes available, we can and should calculate the these chosen parameters (KS and confusion matrix) on a regular base.¹

Secondly, we need an independent opinion. Much like a medical doctor will ask a peer for a second opinion before a risky operation, we need another modeller to look at our model. In a professional setting, this is another team that is specialized in scrutinizing the modelling work of other people. Ideally this team is rather independent and will be a central function, as opposed to the modellers, who should be rather be close to the business.

We do not want to downplay the importance of independent layers (aka “lines of defence”), but for the remainder of this chapter, we will focus on themathematical aspects of measuring the quality of a model and making ...