Designing Machine Learning Systems

Chapter 9. Continual Learning and Test in Production

In Chapter 8, we discussed various ways an ML system can fail in production. We focused on one especially thorny problem that has generated much discussion among both researchers and practitioners: data distribution shifts. We also discussed multiple monitoring techniques and tools to detect data distribution shifts.

This chapter is a continuation of this discussion: how do we adapt our models to data distribution shifts? The answer is by continually updating our ML models. We’ll start with a discussion on what continual learning is and its challenges—spoiler: continual learning is largely an infrastructural problem. Then we’ll lay out a four-stage plan to make continual learning a reality.

After you’ve set up your infrastructure to allow you to update your models as frequently as you want, you might want to consider the question that I’ve been asked by almost every single ML engineer I’ve met: “How often should I retrain my models?” This question is the focus of the next section of the book.

If the model is retrained to adapt to the changing environment, evaluating it on a stationary test set isn’t enough. We’ll cover a seemingly terrifying but necessary concept: test in production. This process is a way to test your systems with live data in production to ensure that your updated model indeed works without catastrophic consequences.

Topics in this chapter and the previous chapter are tightly coupled. Test in production is ...