Machine Learning for Civil and Environmental Engineers

5 Performance Fitness Indicators and Error Metrics

But fairness is squishy and hard to quantify.

Cathy O’Neil¹

Synopsis

In the past few chapters, we learned about data, various ML algorithms, and how we can turn these into ML models. A ML model is declared proper once it satisfies a rigorous training/validation/testing process. We quantify such a process through metrics and indicators. What are these? Why are they needed? And, what makes them a suitable judge of our models? This chapter not only hopes to shed light on the above three questions but aims to go one step further toward showcasing and recommending a series of metrics and indicators ² that could be of interest to some of the problems you will be exploring.

5.1 Introduction

Machine learning, by far, is a data-driven approach. It looks for patterns in data in search of a model³ that can describe the patterns in our data. This implies that a good ML model is one that performs optimally and best describes the phenomenon on hand. The first component (i.e., performs optimally) can be addressed by properly developing and tuning the model. The second component (i.e., best describes the phenomenon on hand) is where we need some form of comparison that contrasts model predictions against a known ground truth. The objective is simple; if the predictions fall in line with an acceptable level of the ground truth, then our model can be declared to have good fitness.⁴

At first glance, the notion of using metrics to quantify ...