book

Machine Learning System Design

Name: Machine Learning System Design
ISBN: 9781633438750

by Arseny Kravchenko, Valerii Babushkin

February 2025

Intermediate to advanced

376 pages

12h 17m

English

Manning Publications

Read now

Unlock full access

Machine Learning System Design
copyright
contents
preface
acknowledgments
about this book
Who should read this book?How this book is organized: A roadmapliveBook discussion forum
about the authors
about the cover illustration
Part 1 Preparations
1 Essentials of machine learning system design
1.1 ML system design: What are you?1.1.1 Why ML system design is so important1.1.2 Roots of ML system design1.2 How this book is structured1.3 When principles of ML system design can be helpfulSummary

2 Is there a problem?
2.1 Problem space vs. solution space2.2 Finding the problem2.2.1 How we can approximate a solution through an ML system2.3 Risks, limitations, and possible consequences2.4 Costs of a mistakeSummary
3 Preliminary research
3.1 What problems can inspire you?3.2 Build or buy: Open source-based or proprietary tech3.2.1 Build or buy3.2.2 Open source-based or proprietary tech3.3 Problem decompositioning3.4 Choosing the right degree of innovation3.4.1 What solutions can be useful?3.4.2 Working on the solution space: Practical exampleSummary
4 Design document
4.1 Common myths surrounding the design document4.1.1 Myth #1. Design documents work only for big companies but not startups4.1.2 Myth #2. Design documents are efficient only for complex projects4.1.3 Myth #3. Every design document should be based on a template4.1.4 Myth #4. Every design document should lead to a deployed system4.2 Goals and antigoals4.3 Design document structure4.4 Reviewing a design document4.4.1 Design document review example4.5 A design doc is a living thingSummary
Part 2 Early stage
5 Loss functions and metrics
5.1 Losses5.1.1 Loss tricks for deep learning models5.2 Metrics5.2.1 Consistency metrics5.2.2 Offline and online metrics, proxy metrics, and hierarchy of metrics5.3 Design document: Adding losses and metrics5.3.1 Metrics and loss functions for Supermegaretail5.3.2 Metrics and loss functions for PhotoStock Inc.5.3.3 Wrap upSummary
6 Gathering datasets
6.1 Data sources6.2 Cooking the dataset6.2.1 ETL6.2.2 Filtering6.2.3 Feature engineering6.2.4 Labeling6.3 Data and metadata6.4 How much data is enough?6.5 Chicken-or-egg problem6.6 Properties of a healthy data pipeline6.7 Design document: Dataset6.7.1 Dataset for Supermegaretail6.7.2 Dataset for PhotoStock Inc.Summary
7 Validation schemas
7.1 Reliable evaluation7.2 Standard schemas7.2.1 Holdout sets7.2.2 Cross-validation7.2.3 The choice of K7.2.4 Time-series validation7.3 Nontrivial schemas7.3.1 Nested validation7.3.2 Adversarial validation7.3.3 Quantifying dataset leakage exploitation7.4 Split updating procedure7.5 Design document: Choosing validation schemas7.5.1 Validation schemas for Supermegaretail7.5.2 Validation schemas for PhotoStock Inc.Summary
8 Baseline solution
8.1 Baseline: What are you?8.2 Constant baselines8.2.1 Why do we need constant baselines?8.3 Model baselines and feature baselines8.4 Variety of deep learning baselines8.5 Baseline comparison8.6 Design document: Baselines8.6.1 Baselines for Supermegaretail8.6.2 Baselines for PhotoStock Inc.Summary
Part 3 Intermediate steps
9 Error analysis
9.1 Learning curve analysis9.1.1 Overfitting and underfitting9.1.2 Loss curve9.1.3 Interpreting loss curves9.1.4 Model-wise learning curve9.1.5 Sample-wise learning curve9.1.6 Double descent9.2 Residual analysis9.2.1 Goals of residual analysis9.2.2 Model assumptions9.2.3 Residual distribution9.2.4 Fairness of residuals9.2.5 Underprediction and overprediction9.2.6 Elasticity curves9.3 Finding commonalities in residuals9.3.1 Worst/best-case analysis9.3.2 Adversarial validation9.3.3 Variety of group analysis9.3.4 Corner-case analysis9.4 Design document: Error analysis9.4.1 Error analysis for Supermegaretail9.4.2 Error analysis for PhotoStock Inc.Summary
10 Training pipelines
10.1 Training pipeline: What are you?10.1.1 Training pipeline vs. inference pipeline10.2 Tools and platforms10.3 Scalability10.4 Configurability10.5 Testing10.5.1 Property-based testing10.6 Design document: Training pipelines10.6.1 Training pipeline for Supermegaretail10.6.2 Training pipeline for PhotoStock Inc.Summary
11 Features and feature engineering
11.1 Feature engineering: What are you?11.1.1 Criteria of good and bad features11.1.2 Feature generation 10111.1.3 Model predictions as a feature11.2 Feature importance analysis11.2.1 Classification of methods11.2.2 Accuracy–interpretability tradeoff11.2.3 Feature importance in deep learning11.3 Feature selection11.3.1 Feature generation vs. feature selection11.3.2 Goals and possible drawbacks11.3.3 Feature selection method overview11.4 Feature store11.4.1 Feature store: Pros and cons11.4.2 Desired properties of a feature store11.4.3 Feature catalog11.5 Design document: Feature engineering11.5.1 Features for Supermegaretail11.5.2 Features for PhotoStock Inc.Summary
12 Measuring and reporting results
12.1 Measuring results12.1.1 Model performance12.1.2 Transition to business metrics12.1.3 Simulated environment12.1.4 Human evaluation12.2 A/B testing12.2.1 Experiment design12.2.2 Splitting strategy12.2.3 Selecting metrics12.2.4 Statistical criteria12.2.5 Simulated experiments12.2.6 When A/B testing is not possible12.3 Reporting results12.3.1 Control and auxiliary metrics12.3.2 Uplift monitoring12.3.3 When to finish the experiment12.3.4 What to report12.3.5 Debrief document12.4 Design document: Measuring and reporting12.4.1 Measuring and reporting for Supermegaretail12.4.2 Measuring and reporting for PhotoStock Inc.Summary
Part 4 Integration and growth
13 Integration
13.1 API design13.1.1 API practices13.2 Release cycle13.3 Operating the system13.3.1 Tech-related connections13.3.2 Non-tech-related connections13.4 Overrides and fallbacks13.5 Design document: Integration13.5.1 Integration for Supermegaretail13.5.2 Integration for PhotoStock Inc.Summary
14 Monitoring and reliability
14.1 Why monitoring is important14.1.1 Incoming data14.1.2 Model14.1.3 Model output14.1.4 Postprocessing/decision-making14.2 Software system health14.3 Data quality and integrity14.3.1 Processing problems14.3.2 Data source corruption14.3.3 Cascade/upstream models14.3.4 Schema change14.3.5 Training-serving skew14.3.6 How to monitor and react14.4 Model quality and relevance14.4.1 Data drift14.4.2 Concept drift14.4.3 How to monitor14.4.4 How to react14.5 Design document: Monitoring14.5.1 Monitoring for Supermegaretail14.5.2 Monitoring for PhotoStock Inc.Summary
15 Serving and inference optimization
15.1 Serving and inference: Challenges15.2 Tradeoffs and patterns15.2.1 Tradeoffs15.2.2 Patterns15.3 Tools and frameworks15.3.1 Choosing a framework15.3.2 Serverless inference15.4 Optimizing inference pipelines15.4.1 Starting with profiling15.4.2 The best optimizing is minimum optimizing15.5 Design document: Serving and inference15.5.1 Serving and inference for Supermegaretail15.5.2 Serving and inference for PhotoStock Inc.Summary
16 Ownership and maintenance
16.1 Accountability16.2 Bus factor16.2.1 Why is being too efficient not beneficial?16.2.2 Why is being too redundant not beneficial?16.2.3 When and how to use the bus factor16.3 Documentation16.4 Complexity16.5 Maintenance and ownership: Supermegaretail and PhotoStock Inc.Summary
index

Content preview from Machine Learning System Design

9 Error analysis

This chapter covers

Learning curve analysis
Residual analysis
Finding commonalities in residuals

Once we’ve assembled the initial building blocks, which include gathering the first dataset, picking the metrics, defining the evaluation procedure, and training the baseline, we are ready to start the iterative adjustments process. Just as backpropagation in neural networks calculates the direction of the fastest loss reduction and passes it backward from layer to layer, error analysis finds the fastest improvement for the whole system.

Error analysis steps up as the compass that guides the iterative updates of your system. It helps you understand the error dynamics during the training phase (learning curve analysis) and ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781633438750Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Machine Learning System Design

by Arseny Kravchenko, Valerii Babushkin