book

Reliable Machine Learning

by Cathy Chen, Niall Richard Murphy, Kranti Parisa, D. Sculley, Todd Underwood

September 2022

Intermediate to advanced

408 pages

12h 49m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Why We Wrote This BookSRE as the Lens on MLIntended AudienceHow This Book Is OrganizedOur ApproachLet’s Knit!Navigating This BookAbout the AuthorsConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgmentsCathy ChenNiall Richard MurphyKranti ParisaD. SculleyTodd Underwood
The ML LifecycleData Collection and AnalysisML Training PipelinesBuild and Validate ApplicationsQuality and Performance EvaluationDefining and Measuring SLOsLaunchMonitoring and Feedback LoopsLessons from the Loop
Data as LiabilityThe Data Sensitivity of ML PipelinesPhases of DataCreationIngestionProcessingStorageManagementAnalysis and VisualizationData ReliabilityDurabilityConsistencyVersion ControlPerformanceAvailabilityData IntegritySecurityPrivacyPolicy and ComplianceConclusion
What Is a Model?A Basic Model Creation WorkflowModel Architecture Versus Model Definition Versus Trained ModelWhere Are the Vulnerabilities?Training DataLabelsTraining MethodsInfrastructure and PipelinesPlatformsFeature GenerationUpgrades and FixesA Set of Useful Questions to Ask About Any ModelAn Example ML SystemYarn Product Click-Prediction ModelFeaturesLabels for FeaturesModel UpdatingModel ServingCommon FailuresConclusion
FeaturesFeature Selection and EngineeringLifecycle of a FeatureFeature SystemsLabelsHuman-Generated LabelsAnnotation WorkforcesMeasuring Human Annotation QualityAn Annotation PlatformActive Learning and AI-Assisted LabelingDocumentation and Training for LabelersMetadataMetadata Systems OverviewDataset MetadataFeature MetadataLabel MetadataPipeline MetadataData Privacy and FairnessPrivacyFairnessConclusion
Evaluating Model ValidityEvaluating Model QualityOffline EvaluationsEvaluation DistributionsA Few Useful MetricsOperationalizing Verification and EvaluationConclusion
Fairness (a.k.a. Fighting Bias)Definitions of FairnessReaching FairnessFairness as a Process Rather than an EndpointA Quick Legal NotePrivacyMethods to Preserve PrivacyA Quick Legal NoteResponsible AIExplanationEffectivenessSocial and Cultural AppropriatenessResponsible AI Along the ML PipelineUse Case BrainstormingData Collection and CleaningModel Creation and TrainingModel Validation and Quality AssessmentModel DeploymentProducts for the MarketConclusion
RequirementsBasic Training System ImplementationFeaturesFeature StoreModel Management SystemOrchestrationQuality EvaluationMonitoringGeneral Reliability PrinciplesMost Failures Will Not Be ML FailuresModels Will Be RetrainedModels Will Have Multiple Versions (at the Same Time!)Good Models Will Become BadData Will Be UnavailableModels Should Be ImprovableFeatures Will Be Added and ChangedModels Can Train Too FastResource Utilization MattersUtilization != EfficiencyOutages Include RecoveryCommon Training Reliability ProblemsData SensitivityExample Data Problem at YarnItReproducibilityExample Reproducibility Problem at YarnItCompute Resource CapacityExample Capacity Problem at YarnItStructural ReliabilityOrganizational ChallengesEthics and Fairness ConsiderationsConclusion
Key Questions for Model ServingWhat Will Be the Load to Our Model?What Are the Prediction Latency Needs of Our Model?Where Does the Model Need to Live?What Are the Hardware Needs for Our Model?How Will the Serving Model Be Stored, Loaded, Versioned, and Updated?What Will Our Feature Pipeline for Serving Look Like?Model Serving ArchitecturesOffline Serving (Batch Inference)Online Serving (Online Inference)Model as a ServiceServing at the EdgeChoosing an ArchitectureModel API DesignTestingServing for Accuracy or Resilience?ScalingAutoscalingCachingDisaster RecoveryEthics and Fairness ConsiderationsConclusion

What Is Production Monitoring and Why Do It?What Does It Look Like?The Concerns That ML Brings to MonitoringReasons for Continual ML Observability—in ProductionProblems with ML Production MonitoringDifficulties of Development Versus ServingA Mindset Change Is RequiredBest Practices for ML Model MonitoringGeneric Pre-serving Model RecommendationsTraining and RetrainingModel Validation (Before Rollout)ServingOther Things to ConsiderHigh-Level Recommendations for Monitoring StrategyConclusion
Anatomy of a Continuous ML SystemTraining ExamplesTraining LabelsFiltering Out Bad DataFeature Stores and Data ManagementUpdating the ModelPushing Updated Models to ServingObservations About Continuous ML SystemsExternal World Events May Influence Our SystemsModels Can Influence Their Own Training DataTemporal Effects Can Arise at Several TimescalesEmergency Response Must Be Done in Real TimeNew Launches Require Staged Ramp-ups and Stable BaselinesModels Must Be Managed Rather Than ShippedContinuous OrganizationsRethinking Noncontinuous ML SystemsConclusion
Incident Management BasicsLife of an IncidentIncident Response RolesAnatomy of an ML-Centric OutageTerminology Reminder: ModelStory TimeStory 1: Searching but Not FindingStory 2: Suddenly Useless PartnersStory 3: Recommend You Find New SuppliersML Incident Management PrinciplesGuiding PrinciplesModel Developer or Data ScientistSoftware EngineerML SRE or Production EngineerProduct Manager or Business LeaderSpecial TopicsProduction Engineers and ML Engineering Versus ModelingThe Ethical On-Call Engineer ManifestoConclusion
Different Types of ProductsAgile ML?ML Product Development PhasesDiscovery and DefinitionBusiness Goal SettingMVP Construction and ValidationModel and Product DevelopmentDeploymentSupport and MaintenanceBuild Versus BuyModelsData Processing InfrastructureEnd-to-End PlatformsScoring Approach for Making the DecisionMaking the DecisionSample YarnIt Store Features Powered by MLShowcasing Popular Yarns by Total SalesRecommendations Based on Browsing HistoryCross-selling and UpsellingContent-Based FilteringCollaborative FilteringConclusion
Chapter AssumptionsLeader-Based ViewpointDetail MattersML Needs to Know About the BusinessThe Most Important Assumption You MakeThe Value of MLSignificant Organizational RisksML Is Not MagicMental (Way of Thinking) Model InertiaSurfacing Risk Correctly in Different CulturesSiloed Teams Don’t Solve All ProblemsImplementation ModelsRemembering the GoalGreenfield Versus BrownfieldML Roles and ResponsibilitiesHow to Hire ML FolksOrganizational Design and IncentivesStrategyStructureProcessesRewardsPeopleA Note on SequencingConclusion
Scenario 1: A New Centralized ML TeamBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationScenario 2: Decentralized ML Infrastructure and ExpertiseBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationScenario 3: Hybrid with Centralized Infrastructure/Decentralized ModelingBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationConclusion
1. Accommodating Privacy and Data Retention Policies in ML PipelinesBackgroundProblem and ResolutionTakeaways2. Continuous ML Model Impacting TrafficBackgroundProblem and ResolutionTakeaways3. Steel InspectionBackgroundProblem and ResolutionTakeaways4. NLP MLOps: Profiling and Staging Load TestBackgroundProblem and ResolutionTakeaways5. Ad Click Prediction: Databases Versus RealityBackgroundProblem and ResolutionTakeaways6. Testing and Measuring Dependencies in ML WorkflowBackgroundProblem and ResolutionTakeaways

Content preview from Reliable Machine Learning

Preface

This is not a book about how machine learning works. This is a book about how to make machine learning work—for you.

The way that machine learning (ML) works is fascinating. The math, algorithms, and statistical insights that surround and support ML are themselves of interest, and what they can achieve when applied to the right data can be nothing short of magical. But we do something a little different in this book. We are not algorithm oriented—we are whole-system oriented. In short, we talk about everything other than the algorithms. Plenty of other works cover the algorithmic component of ML in great detail, but this one is deliberately focused on the whole lifecycle of ML, giving it the time and attention it doesn’t really get elsewhere.

This means that we talk about the messy, complicated, and occasionally frustrating work involved in shepherding data correctly and responsibly; reliable model building; ensuring a smooth (and reversible) path to production; safety in updating; and concerns about cost, performance, business goals, and organizational structure. We attempt to cover everything involved in having ML happen reliably in your organization.

Why We Wrote This Book

We firmly believe at least some of the hype: ML and AI techniques are currently reshaping computing and society at an accelerating rate. To that extent, the public hype has not caught up with the private reality in some respects.¹ But we are also grounded and experienced enough to understand just ...