book

Reliable Machine Learning

by Cathy Chen, Niall Richard Murphy, Kranti Parisa, D. Sculley, Todd Underwood

September 2022

Intermediate to advanced

408 pages

12h 49m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Why We Wrote This BookSRE as the Lens on MLIntended AudienceHow This Book Is OrganizedOur ApproachLet’s Knit!Navigating This BookAbout the AuthorsConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgmentsCathy ChenNiall Richard MurphyKranti ParisaD. SculleyTodd Underwood
The ML LifecycleData Collection and AnalysisML Training PipelinesBuild and Validate ApplicationsQuality and Performance EvaluationDefining and Measuring SLOsLaunchMonitoring and Feedback LoopsLessons from the Loop
Data as LiabilityThe Data Sensitivity of ML PipelinesPhases of DataCreationIngestionProcessingStorageManagementAnalysis and VisualizationData ReliabilityDurabilityConsistencyVersion ControlPerformanceAvailabilityData IntegritySecurityPrivacyPolicy and ComplianceConclusion
What Is a Model?A Basic Model Creation WorkflowModel Architecture Versus Model Definition Versus Trained ModelWhere Are the Vulnerabilities?Training DataLabelsTraining MethodsInfrastructure and PipelinesPlatformsFeature GenerationUpgrades and FixesA Set of Useful Questions to Ask About Any ModelAn Example ML SystemYarn Product Click-Prediction ModelFeaturesLabels for FeaturesModel UpdatingModel ServingCommon FailuresConclusion
FeaturesFeature Selection and EngineeringLifecycle of a FeatureFeature SystemsLabelsHuman-Generated LabelsAnnotation WorkforcesMeasuring Human Annotation QualityAn Annotation PlatformActive Learning and AI-Assisted LabelingDocumentation and Training for LabelersMetadataMetadata Systems OverviewDataset MetadataFeature MetadataLabel MetadataPipeline MetadataData Privacy and FairnessPrivacyFairnessConclusion
Evaluating Model ValidityEvaluating Model QualityOffline EvaluationsEvaluation DistributionsA Few Useful MetricsOperationalizing Verification and EvaluationConclusion
Fairness (a.k.a. Fighting Bias)Definitions of FairnessReaching FairnessFairness as a Process Rather than an EndpointA Quick Legal NotePrivacyMethods to Preserve PrivacyA Quick Legal NoteResponsible AIExplanationEffectivenessSocial and Cultural AppropriatenessResponsible AI Along the ML PipelineUse Case BrainstormingData Collection and CleaningModel Creation and TrainingModel Validation and Quality AssessmentModel DeploymentProducts for the MarketConclusion
RequirementsBasic Training System ImplementationFeaturesFeature StoreModel Management SystemOrchestrationQuality EvaluationMonitoringGeneral Reliability PrinciplesMost Failures Will Not Be ML FailuresModels Will Be RetrainedModels Will Have Multiple Versions (at the Same Time!)Good Models Will Become BadData Will Be UnavailableModels Should Be ImprovableFeatures Will Be Added and ChangedModels Can Train Too FastResource Utilization MattersUtilization != EfficiencyOutages Include RecoveryCommon Training Reliability ProblemsData SensitivityExample Data Problem at YarnItReproducibilityExample Reproducibility Problem at YarnItCompute Resource CapacityExample Capacity Problem at YarnItStructural ReliabilityOrganizational ChallengesEthics and Fairness ConsiderationsConclusion
Key Questions for Model ServingWhat Will Be the Load to Our Model?What Are the Prediction Latency Needs of Our Model?Where Does the Model Need to Live?What Are the Hardware Needs for Our Model?How Will the Serving Model Be Stored, Loaded, Versioned, and Updated?What Will Our Feature Pipeline for Serving Look Like?Model Serving ArchitecturesOffline Serving (Batch Inference)Online Serving (Online Inference)Model as a ServiceServing at the EdgeChoosing an ArchitectureModel API DesignTestingServing for Accuracy or Resilience?ScalingAutoscalingCachingDisaster RecoveryEthics and Fairness ConsiderationsConclusion

What Is Production Monitoring and Why Do It?What Does It Look Like?The Concerns That ML Brings to MonitoringReasons for Continual ML Observability—in ProductionProblems with ML Production MonitoringDifficulties of Development Versus ServingA Mindset Change Is RequiredBest Practices for ML Model MonitoringGeneric Pre-serving Model RecommendationsTraining and RetrainingModel Validation (Before Rollout)ServingOther Things to ConsiderHigh-Level Recommendations for Monitoring StrategyConclusion
Anatomy of a Continuous ML SystemTraining ExamplesTraining LabelsFiltering Out Bad DataFeature Stores and Data ManagementUpdating the ModelPushing Updated Models to ServingObservations About Continuous ML SystemsExternal World Events May Influence Our SystemsModels Can Influence Their Own Training DataTemporal Effects Can Arise at Several TimescalesEmergency Response Must Be Done in Real TimeNew Launches Require Staged Ramp-ups and Stable BaselinesModels Must Be Managed Rather Than ShippedContinuous OrganizationsRethinking Noncontinuous ML SystemsConclusion
Incident Management BasicsLife of an IncidentIncident Response RolesAnatomy of an ML-Centric OutageTerminology Reminder: ModelStory TimeStory 1: Searching but Not FindingStory 2: Suddenly Useless PartnersStory 3: Recommend You Find New SuppliersML Incident Management PrinciplesGuiding PrinciplesModel Developer or Data ScientistSoftware EngineerML SRE or Production EngineerProduct Manager or Business LeaderSpecial TopicsProduction Engineers and ML Engineering Versus ModelingThe Ethical On-Call Engineer ManifestoConclusion
Different Types of ProductsAgile ML?ML Product Development PhasesDiscovery and DefinitionBusiness Goal SettingMVP Construction and ValidationModel and Product DevelopmentDeploymentSupport and MaintenanceBuild Versus BuyModelsData Processing InfrastructureEnd-to-End PlatformsScoring Approach for Making the DecisionMaking the DecisionSample YarnIt Store Features Powered by MLShowcasing Popular Yarns by Total SalesRecommendations Based on Browsing HistoryCross-selling and UpsellingContent-Based FilteringCollaborative FilteringConclusion
Chapter AssumptionsLeader-Based ViewpointDetail MattersML Needs to Know About the BusinessThe Most Important Assumption You MakeThe Value of MLSignificant Organizational RisksML Is Not MagicMental (Way of Thinking) Model InertiaSurfacing Risk Correctly in Different CulturesSiloed Teams Don’t Solve All ProblemsImplementation ModelsRemembering the GoalGreenfield Versus BrownfieldML Roles and ResponsibilitiesHow to Hire ML FolksOrganizational Design and IncentivesStrategyStructureProcessesRewardsPeopleA Note on SequencingConclusion
Scenario 1: A New Centralized ML TeamBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationScenario 2: Decentralized ML Infrastructure and ExpertiseBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationScenario 3: Hybrid with Centralized Infrastructure/Decentralized ModelingBackground and Organizational DescriptionProcessRewardsPeopleDefault ImplementationConclusion
1. Accommodating Privacy and Data Retention Policies in ML PipelinesBackgroundProblem and ResolutionTakeaways2. Continuous ML Model Impacting TrafficBackgroundProblem and ResolutionTakeaways3. Steel InspectionBackgroundProblem and ResolutionTakeaways4. NLP MLOps: Profiling and Staging Load TestBackgroundProblem and ResolutionTakeaways5. Ad Click Prediction: Databases Versus RealityBackgroundProblem and ResolutionTakeaways6. Testing and Measuring Dependencies in ML WorkflowBackgroundProblem and ResolutionTakeaways

Content preview from Reliable Machine Learning

Chapter 12. How Product and ML Interact

As companies rush to use ML capabilities to meet customer needs, they are eager to leverage cutting-edge research to tackle a wide variety of business applications. Many of the product teams and business managers, still anchored in traditional software product development methodologies, find themselves in a new and unfamiliar territory: building ML products.

Building your first ML product can be overwhelming. It’s not just a question of getting ML right, difficult enough in itself; rather, the integration of the ML into the rest of the product (and the rest of the business) requires many things that need to work together. Among these, data collection practices and governance, the quality of data, definition of product behavior, UI/UX, and business goals all contribute to the success of an ML-based product or feature.

Different Types of Products

One of the important and useful features of ML is that it can be applied to many types of products. It can be used in analytics applications to derive insights about business trends and metrics. It can be rolled into an appliance or device to be shipped to consumers. Sophisticated ML systems are built into self-driving cars to detect other objects and to make decisions about driving. The breadth of ML applications is huge and growing. As a result of this huge diversity of use cases, organizations focused on integrating ML into their existing or new products face an extremely steep learning curve ...