Reliable Machine Learning

Chapter 1. Introduction

We begin with a model, or framework, for adding machine learning (ML) to a website, widely applicable across a number of domains—not just this example. This model we call the ML loop.

The ML Lifecycle

ML applications are never really done. They also don’t start or stop in any one place, either technically or organizationally. ML model developers often hope their lives will be simple, and they’ll have to collect data and train a model only once, but it rarely happens that way.

A simple thought experiment can help us understand why. Suppose we have an ML model, and we are investigating whether the model works well enough (according to a certain threshold) or doesn’t. If it doesn’t work well enough, data scientists, business analysts, and ML engineers will typically collaborate on how to understand the failures and improve upon them. This involves, as you might expect, a lot of work: perhaps modifying the existing training pipeline to change some features, adding or removing some data, and restructuring the model in order to iterate on what has already been done.

Conversely, if the model is working well, what usually happens is that organizations get excited. The natural thought is that if we can make so much progress with one, naïve attempt, imagine how much better we can do if we work harder on it and get more sophisticated. This typically involves—you guessed it—modifying the existing training pipeline, changing features, adding or removing data, and possibly ...