book

Designing Machine Learning Systems

by Chip Huyen

May 2022

Intermediate to advanced

386 pages

12h 25m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who This Book Is ForWhat This Book Is NotNavigating This BookGitHub Repository and CommunityConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
When to Use Machine LearningMachine Learning Use CasesUnderstanding Machine Learning SystemsMachine Learning in Research Versus in ProductionMachine Learning Systems Versus Traditional SoftwareSummary
Business and ML ObjectivesRequirements for ML SystemsReliabilityScalabilityMaintainabilityAdaptabilityIterative ProcessFraming ML ProblemsTypes of ML TasksObjective FunctionsMind Versus DataSummary
Data SourcesData FormatsJSONRow-Major Versus Column-Major FormatText Versus Binary FormatData ModelsRelational ModelNoSQLStructured Versus Unstructured DataData Storage Engines and ProcessingTransactional and Analytical ProcessingETL: Extract, Transform, and LoadModes of DataflowData Passing Through DatabasesData Passing Through ServicesData Passing Through Real-Time TransportBatch Processing Versus Stream ProcessingSummary
SamplingNonprobability SamplingSimple Random SamplingStratified SamplingWeighted SamplingReservoir SamplingImportance SamplingLabelingHand LabelsNatural LabelsHandling the Lack of LabelsClass ImbalanceChallenges of Class ImbalanceHandling Class ImbalanceData AugmentationSimple Label-Preserving TransformationsPerturbationData SynthesisSummary
Learned Features Versus Engineered FeaturesCommon Feature Engineering OperationsHandling Missing ValuesScalingDiscretizationEncoding Categorical FeaturesFeature CrossingDiscrete and Continuous Positional EmbeddingsData LeakageCommon Causes for Data LeakageDetecting Data LeakageEngineering Good FeaturesFeature ImportanceFeature GeneralizationSummary
Model Development and TrainingEvaluating ML ModelsEnsemblesExperiment Tracking and VersioningDistributed TrainingAutoMLModel Offline EvaluationBaselinesEvaluation MethodsSummary
Machine Learning Deployment MythsMyth 1: You Only Deploy One or Two ML Models at a TimeMyth 2: If We Don’t Do Anything, Model Performance Remains the SameMyth 3: You Won’t Need to Update Your Models as MuchMyth 4: Most ML Engineers Don’t Need to Worry About ScaleBatch Prediction Versus Online PredictionFrom Batch Prediction to Online PredictionUnifying Batch Pipeline and Streaming PipelineModel CompressionLow-Rank FactorizationKnowledge DistillationPruningQuantizationML on the Cloud and on the EdgeCompiling and Optimizing Models for Edge DevicesML in BrowsersSummary
Causes of ML System FailuresSoftware System FailuresML-Specific FailuresData Distribution ShiftsTypes of Data Distribution ShiftsGeneral Data Distribution ShiftsDetecting Data Distribution ShiftsAddressing Data Distribution ShiftsMonitoring and ObservabilityML-Specific MetricsMonitoring ToolboxObservabilitySummary
Continual LearningStateless Retraining Versus Stateful TrainingWhy Continual Learning?Continual Learning ChallengesFour Stages of Continual LearningHow Often to Update Your ModelsTest in ProductionShadow DeploymentA/B TestingCanary ReleaseInterleaving ExperimentsBanditsSummary

Storage and ComputePublic Cloud Versus Private Data CentersDevelopment EnvironmentDev Environment SetupStandardizing Dev EnvironmentsFrom Dev to Prod: ContainersResource ManagementCron, Schedulers, and OrchestratorsData Science Workflow ManagementML PlatformModel DeploymentModel StoreFeature StoreBuild Versus BuySummary
User ExperienceEnsuring User Experience ConsistencyCombatting “Mostly Correct” PredictionsSmooth FailingTeam StructureCross-functional Teams CollaborationEnd-to-End Data ScientistsResponsible AIIrresponsible AI: Case StudiesA Framework for Responsible AISummary

Content preview from Designing Machine Learning Systems

Chapter 1. Overview of Machine Learning Systems

In November 2016, Google announced that it had incorporated its multilingual neural machine translation system into Google Translate, marking one of the first success stories of deep artificial neural networks in production at scale.¹ According to Google, with this update, the quality of translation improved more in a single leap than they had seen in the previous 10 years combined.

This success of deep learning renewed the interest in machine learning (ML) at large. Since then, more and more companies have turned toward ML for solutions to their most challenging problems. In just five years, ML has found its way into almost every aspect of our lives: how we access information, how we communicate, how we work, how we find love. The spread of ML has been so rapid that it’s already hard to imagine life without it. Yet there are still many more use cases for ML waiting to be explored in fields such as health care, transportation, farming, and even in helping us understand the universe.²

Many people, when they hear “machine learning system,” think of just the ML algorithms being used such as logistic regression or different types of neural networks. However, the algorithm is only a small part of an ML system in production. The system also includes the business requirements that gave birth to the ML project in the first place, the interface where users and developers interact with your system, the data stack, and the logic for developing, ...