book

Reliable Machine Learning

by Cathy Chen, Niall Richard Murphy, Kranti Parisa, D. Sculley, Todd Underwood

September 2022

Intermediate to advanced

408 pages

12h 49m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Why We Wrote This BookSRE as the Lens on MLIntended AudienceHow This Book Is OrganizedOur ApproachLet’s Knit!Navigating This BookAbout the AuthorsConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgmentsCathy ChenNiall Richard MurphyKranti ParisaD. SculleyTodd Underwood
The ML LifecycleData Collection and AnalysisML Training PipelinesBuild and Validate ApplicationsQuality and Performance EvaluationDefining and Measuring SLOsLaunchMonitoring and Feedback LoopsLessons from the Loop
Data as LiabilityThe Data Sensitivity of ML PipelinesPhases of DataCreationIngestionProcessingStorageManagementAnalysis and VisualizationData ReliabilityDurabilityConsistencyVersion ControlPerformanceAvailabilityData IntegritySecurityPrivacyPolicy and ComplianceConclusion
What Is a Model?A Basic Model Creation WorkflowModel Architecture Versus Model Definition Versus Trained ModelWhere Are the Vulnerabilities?Training DataLabelsTraining MethodsInfrastructure and PipelinesPlatformsFeature GenerationUpgrades and FixesA Set of Useful Questions to Ask About Any ModelAn Example ML SystemYarn Product Click-Prediction ModelFeaturesLabels for FeaturesModel UpdatingModel ServingCommon FailuresConclusion
FeaturesFeature Selection and EngineeringLifecycle of a FeatureFeature SystemsLabelsHuman-Generated LabelsAnnotation WorkforcesMeasuring Human Annotation QualityAn Annotation PlatformActive Learning and AI-Assisted LabelingDocumentation and Training for LabelersMetadataMetadata Systems OverviewDataset MetadataFeature MetadataLabel MetadataPipeline MetadataData Privacy and FairnessPrivacyFairnessConclusion
Evaluating Model ValidityEvaluating Model QualityOffline EvaluationsEvaluation DistributionsA Few Useful MetricsOperationalizing Verification and EvaluationConclusion
Fairness (a.k.a. Fighting Bias)Definitions of FairnessReaching FairnessFairness as a Process Rather than an EndpointA Quick Legal NotePrivacyMethods to Preserve PrivacyA Quick Legal NoteResponsible AIExplanationEffectivenessSocial and Cultural AppropriatenessResponsible AI Along the ML PipelineUse Case BrainstormingData Collection and CleaningModel Creation and TrainingModel Validation and Quality AssessmentModel DeploymentProducts for the MarketConclusion
RequirementsBasic Training System ImplementationFeaturesFeature StoreModel Management SystemOrchestrationQuality EvaluationMonitoringGeneral Reliability PrinciplesMost Failures Will Not Be ML FailuresModels Will Be RetrainedModels Will Have Multiple Versions (at the Same Time!)Good Models Will Become BadData Will Be UnavailableModels Should Be ImprovableFeatures Will Be Added and ChangedModels Can Train Too FastResource Utilization MattersUtilization != EfficiencyOutages Include RecoveryCommon Training Reliability ProblemsData SensitivityExample Data Problem at YarnItReproducibilityExample Reproducibility Problem at YarnItCompute Resource CapacityExample Capacity Problem at YarnItStructural ReliabilityOrganizational ChallengesEthics and Fairness ConsiderationsConclusion
Key Questions for Model ServingWhat Will Be the Load to Our Model?What Are the Prediction Latency Needs of Our Model?Where Does the Model Need to Live?What Are the Hardware Needs for Our Model?How Will the Serving Model Be Stored, Loaded, Versioned, and Updated?What Will Our Feature Pipeline for Serving Look Like?Model Serving ArchitecturesOffline Serving (Batch Inference)Online Serving (Online Inference)Model as a ServiceServing at the EdgeChoosing an ArchitectureModel API DesignTestingServing for Accuracy or Resilience?ScalingAutoscalingCachingDisaster RecoveryEthics and Fairness ConsiderationsConclusion

What Is Production Monitoring and Why Do It?What Does It Look Like?The Concerns That ML Brings to MonitoringReasons for Continual ML Observability—in ProductionProblems with ML Production MonitoringDifficulties of Development Versus ServingA Mindset Change Is RequiredBest Practices for ML Model MonitoringGeneric Pre-serving Model RecommendationsTraining and RetrainingModel Validation (Before Rollout)ServingOther Things to ConsiderHigh-Level Recommendations for Monitoring StrategyConclusion
Anatomy of a Continuous ML SystemTraining ExamplesTraining LabelsFiltering Out Bad DataFeature Stores and Data ManagementUpdating the ModelPushing Updated Models to ServingObservations About Continuous ML SystemsExternal World Events May Influence Our SystemsModels Can Influence Their Own Training DataTemporal Effects Can Arise at Several TimescalesEmergency Response Must Be Done in Real TimeNew Launches Require Staged Ramp-ups and Stable BaselinesModels Must Be Managed Rather Than ShippedContinuous OrganizationsRethinking Noncontinuous ML SystemsConclusion
Incident Management BasicsLife of an IncidentIncident Response RolesAnatomy of an ML-Centric OutageTerminology Reminder: ModelStory TimeStory 1: Searching but Not FindingStory 2: Suddenly Useless PartnersStory 3: Recommend You Find New SuppliersML Incident Management PrinciplesGuiding PrinciplesModel Developer or Data ScientistSoftware EngineerML SRE or Production EngineerProduct Manager or Business LeaderSpecial TopicsProduction Engineers and ML Engineering Versus ModelingThe Ethical On-Call Engineer ManifestoConclusion
Different Types of ProductsAgile ML?ML Product Development PhasesDiscovery and DefinitionBusiness Goal SettingMVP Construction and ValidationModel and Product DevelopmentDeploymentSupport and MaintenanceBuild Versus BuyModelsData Processing InfrastructureEnd-to-End PlatformsScoring Approach for Making the DecisionMaking the DecisionSample YarnIt Store Features Powered by MLShowcasing Popular Yarns by Total SalesRecommendations Based on Browsing HistoryCross-selling and UpsellingContent-Based FilteringCollaborative FilteringConclusion
Chapter AssumptionsLeader-Based ViewpointDetail MattersML Needs to Know About the BusinessThe Most Important Assumption You MakeThe Value of MLSignificant Organizational RisksML Is Not MagicMental (Way of Thinking) Model InertiaSurfacing Risk Correctly in Different CulturesSiloed Teams Don’t Solve All ProblemsImplementation ModelsRemembering the GoalGreenfield Versus BrownfieldML Roles and ResponsibilitiesHow to Hire ML FolksOrganizational Design and IncentivesStrategyStructureProcessesRewardsPeopleA Note on SequencingConclusion
Scenario 1: A New Centralized ML TeamBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationScenario 2: Decentralized ML Infrastructure and ExpertiseBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationScenario 3: Hybrid with Centralized Infrastructure/Decentralized ModelingBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationConclusion
1. Accommodating Privacy and Data Retention Policies in ML PipelinesBackgroundProblem and ResolutionTakeaways2. Continuous ML Model Impacting TrafficBackgroundProblem and ResolutionTakeaways3. Steel InspectionBackgroundProblem and ResolutionTakeaways4. NLP MLOps: Profiling and Staging Load TestBackgroundProblem and ResolutionTakeaways5. Ad Click Prediction: Databases Versus RealityBackgroundProblem and ResolutionTakeaways6. Testing and Measuring Dependencies in ML WorkflowBackgroundProblem and ResolutionTakeaways

Content preview from Reliable Machine Learning

Chapter 8. Serving

You’ve made a model; now you have to get it out there into the world and start predicting things. This is a process often known as serving the model. That’s a common shorthand for “creating a structure to ensure our system can ask the model to make predictions on new examples, and return those predictions to the people or systems that need them” (so you can see why the shorthand was invented).

In our yarnit.ai online store example, we can imagine that our team has just created a model that is great at predicting the likelihood that a given user will purchase a given product. We need to have a way for the model to share its predictions with our overall system. But how, exactly, should we set this up?

We have a range of possibilities, each with different architectures and trade-offs. They are sufficiently different in approach that it might not be obvious looking at the list that these are all attempts to solve the same problem: how can we integrate our predictions with the overall system? We could do any of the following:

Load the model into 1,000 servers in Des Moines, Iowa, and feed all incoming traffic to these servers.
Precompute the model’s predictions for the 100,000,000 most commonly seen combinations of yarn products and user queries using a big offline batch-processing job. Write those to a shared database once a day that is read by our system, and use a default score of p = 0.01 for anything not in that list.
Create a JavaScript version of the model ...