book

Machine Learning System Design

by Arseny Kravchenko, Valerii Babushkin

February 2025

Intermediate to advanced

376 pages

12h 17m

English

Manning Publications

Read now

Unlock full access

Who should read this book?How this book is organized: A roadmapliveBook discussion forum
1.1 ML system design: What are you?1.1.1 Why ML system design is so important1.1.2 Roots of ML system design1.2 How this book is structured1.3 When principles of ML system design can be helpfulSummary

2.1 Problem space vs. solution space2.2 Finding the problem2.2.1 How we can approximate a solution through an ML system2.3 Risks, limitations, and possible consequences2.4 Costs of a mistakeSummary
3.1 What problems can inspire you?3.2 Build or buy: Open source-based or proprietary tech3.2.1 Build or buy3.2.2 Open source-based or proprietary tech3.3 Problem decompositioning3.4 Choosing the right degree of innovation3.4.1 What solutions can be useful?3.4.2 Working on the solution space: Practical exampleSummary
4.1 Common myths surrounding the design document4.1.1 Myth #1. Design documents work only for big companies but not startups4.1.2 Myth #2. Design documents are efficient only for complex projects4.1.3 Myth #3. Every design document should be based on a template4.1.4 Myth #4. Every design document should lead to a deployed system4.2 Goals and antigoals4.3 Design document structure4.4 Reviewing a design document4.4.1 Design document review example4.5 A design doc is a living thingSummary
5.1 Losses5.1.1 Loss tricks for deep learning models5.2 Metrics5.2.1 Consistency metrics5.2.2 Offline and online metrics, proxy metrics, and hierarchy of metrics5.3 Design document: Adding losses and metrics5.3.1 Metrics and loss functions for Supermegaretail5.3.2 Metrics and loss functions for PhotoStock Inc.5.3.3 Wrap upSummary
6.1 Data sources6.2 Cooking the dataset6.2.1 ETL6.2.2 Filtering6.2.3 Feature engineering6.2.4 Labeling6.3 Data and metadata6.4 How much data is enough?6.5 Chicken-or-egg problem6.6 Properties of a healthy data pipeline6.7 Design document: Dataset6.7.1 Dataset for Supermegaretail6.7.2 Dataset for PhotoStock Inc.Summary
7.1 Reliable evaluation7.2 Standard schemas7.2.1 Holdout sets7.2.2 Cross-validation7.2.3 The choice of K7.2.4 Time-series validation7.3 Nontrivial schemas7.3.1 Nested validation7.3.2 Adversarial validation7.3.3 Quantifying dataset leakage exploitation7.4 Split updating procedure7.5 Design document: Choosing validation schemas7.5.1 Validation schemas for Supermegaretail7.5.2 Validation schemas for PhotoStock Inc.Summary
8.1 Baseline: What are you?8.2 Constant baselines8.2.1 Why do we need constant baselines?8.3 Model baselines and feature baselines8.4 Variety of deep learning baselines8.5 Baseline comparison8.6 Design document: Baselines8.6.1 Baselines for Supermegaretail8.6.2 Baselines for PhotoStock Inc.Summary
9.1 Learning curve analysis9.1.1 Overfitting and underfitting9.1.2 Loss curve9.1.3 Interpreting loss curves9.1.4 Model-wise learning curve9.1.5 Sample-wise learning curve9.1.6 Double descent9.2 Residual analysis9.2.1 Goals of residual analysis9.2.2 Model assumptions9.2.3 Residual distribution9.2.4 Fairness of residuals9.2.5 Underprediction and overprediction9.2.6 Elasticity curves9.3 Finding commonalities in residuals9.3.1 Worst/best-case analysis9.3.2 Adversarial validation9.3.3 Variety of group analysis9.3.4 Corner-case analysis9.4 Design document: Error analysis9.4.1 Error analysis for Supermegaretail9.4.2 Error analysis for PhotoStock Inc.Summary
10.1 Training pipeline: What are you?10.1.1 Training pipeline vs. inference pipeline10.2 Tools and platforms10.3 Scalability10.4 Configurability10.5 Testing10.5.1 Property-based testing10.6 Design document: Training pipelines10.6.1 Training pipeline for Supermegaretail10.6.2 Training pipeline for PhotoStock Inc.Summary
11.1 Feature engineering: What are you?11.1.1 Criteria of good and bad features11.1.2 Feature generation 10111.1.3 Model predictions as a feature11.2 Feature importance analysis11.2.1 Classification of methods11.2.2 Accuracy–interpretability tradeoff11.2.3 Feature importance in deep learning11.3 Feature selection11.3.1 Feature generation vs. feature selection11.3.2 Goals and possible drawbacks11.3.3 Feature selection method overview11.4 Feature store11.4.1 Feature store: Pros and cons11.4.2 Desired properties of a feature store11.4.3 Feature catalog11.5 Design document: Feature engineering11.5.1 Features for Supermegaretail11.5.2 Features for PhotoStock Inc.Summary
12.1 Measuring results12.1.1 Model performance12.1.2 Transition to business metrics12.1.3 Simulated environment12.1.4 Human evaluation12.2 A/B testing12.2.1 Experiment design12.2.2 Splitting strategy12.2.3 Selecting metrics12.2.4 Statistical criteria12.2.5 Simulated experiments12.2.6 When A/B testing is not possible12.3 Reporting results12.3.1 Control and auxiliary metrics12.3.2 Uplift monitoring12.3.3 When to finish the experiment12.3.4 What to report12.3.5 Debrief document12.4 Design document: Measuring and reporting12.4.1 Measuring and reporting for Supermegaretail12.4.2 Measuring and reporting for PhotoStock Inc.Summary
13.1 API design13.1.1 API practices13.2 Release cycle13.3 Operating the system13.3.1 Tech-related connections13.3.2 Non-tech-related connections13.4 Overrides and fallbacks13.5 Design document: Integration13.5.1 Integration for Supermegaretail13.5.2 Integration for PhotoStock Inc.Summary
14.1 Why monitoring is important14.1.1 Incoming data14.1.2 Model14.1.3 Model output14.1.4 Postprocessing/decision-making14.2 Software system health14.3 Data quality and integrity14.3.1 Processing problems14.3.2 Data source corruption14.3.3 Cascade/upstream models14.3.4 Schema change14.3.5 Training-serving skew14.3.6 How to monitor and react14.4 Model quality and relevance14.4.1 Data drift14.4.2 Concept drift14.4.3 How to monitor14.4.4 How to react14.5 Design document: Monitoring14.5.1 Monitoring for Supermegaretail14.5.2 Monitoring for PhotoStock Inc.Summary
15.1 Serving and inference: Challenges15.2 Tradeoffs and patterns15.2.1 Tradeoffs15.2.2 Patterns15.3 Tools and frameworks15.3.1 Choosing a framework15.3.2 Serverless inference15.4 Optimizing inference pipelines15.4.1 Starting with profiling15.4.2 The best optimizing is minimum optimizing15.5 Design document: Serving and inference15.5.1 Serving and inference for Supermegaretail15.5.2 Serving and inference for PhotoStock Inc.Summary
16.1 Accountability16.2 Bus factor16.2.1 Why is being too efficient not beneficial?16.2.2 Why is being too redundant not beneficial?16.2.3 When and how to use the bus factor16.3 Documentation16.4 Complexity16.5 Maintenance and ownership: Supermegaretail and PhotoStock Inc.Summary

Content preview from Machine Learning System Design

5 Loss functions and metrics

This chapter covers

Selecting proper metrics and losses for your machine learning system
Defining and utilizing proxy metrics
Applying the hierarchy of metrics

In the previous chapter, we first touched on the topic of creating a design document for your machine learning (ML) system. We figured out why a design document is subject to constant edits and why all the changes you implement in it are not only inevitable but also necessary.

Unfortunately, an ML system can’t directly solve a problem, but it can try to approximate it by optimizing a specific task. To do that efficiently, it must be adjusted, appropriately guided, and monitored.

To direct an ML system’s effort, we use its algorithm’s loss function ...