book

Machine Learning Production Systems

by Robert Crowe, Hannes Hapke, Emily Caveness, Di Zhu

October 2024

Beginner to intermediate

474 pages

13h 56m

English

O'Reilly Media, Inc.

Audio summary available

Read now

Unlock full access

Includes

Quizzes

Foreword
Preface
Who Should Read This BookWhy We Wrote This BookNavigating This BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsRobertHannesEmilyDi
1. Introduction to Machine Learning Production Systems
What Is Production Machine Learning?Benefits of Machine Learning PipelinesFocus on Developing New Models, Not on Maintaining Existing ModelsPrevention of BugsCreation of Records for Debugging and Reproducing ResultsStandardizationThe Business Case for ML Pipelines When to Use Machine Learning PipelinesSteps in a Machine Learning PipelineData Ingestion and Data VersioningData ValidationFeature EngineeringModel Training and Model TuningModel AnalysisModel DeploymentLooking Ahead
2. Collecting, Labeling, and Validating Data
Important Considerations in Data CollectionResponsible Data CollectionLabeling Data: Data Changes and Drift in Production MLLabeling Data: Direct Labeling and Human LabelingValidating Data: Detecting Data IssuesValidating Data: TensorFlow Data ValidationSkew Detection with TFDVTypes of SkewExample: Spotting Imbalanced Datasets with TensorFlow Data ValidationConclusion
3. Feature Engineering and Feature Selection
Introduction to Feature EngineeringPreprocessing OperationsFeature Engineering TechniquesNormalizing and StandardizingBucketizingFeature CrossesDimensionality and EmbeddingsVisualizationFeature Transformation at ScaleChoose a Framework That Scales WellAvoid Training–Serving SkewConsider Instance-Level Versus Full-Pass TransformationsUsing TensorFlow TransformAnalyzersCode ExampleFeature SelectionFeature SpacesFeature Selection OverviewFilter MethodsWrapper MethodsEmbedded MethodsFeature and Example Selection for LLMs and GenAIExample: Using TF Transform to Tokenize TextBenefits of Using TF Transform Alternatives to TF TransformConclusion
4. Data Journey and Data Storage
Data JourneyML MetadataUsing a SchemaSchema DevelopmentSchema EnvironmentsChanges Across Datasets Enterprise Data StorageFeature StoresData WarehousesData LakesConclusion
5. Advanced Labeling, Augmentation, and Data Preprocessing
Advanced LabelingSemi-Supervised LabelingActive LearningWeak SupervisionAdvanced Labeling ReviewData AugmentationExample: CIFAR-10Other Augmentation TechniquesData Augmentation ReviewPreprocessing Time Series Data: An ExampleWindowingSamplingConclusion
6. Model Resource Management Techniques
Dimensionality Reduction: Dimensionality Effect on PerformanceExample: Word Embedding Using KerasCurse of DimensionalityAdding Dimensions Increases Feature Space VolumeDimensionality ReductionQuantization and PruningMobile, IoT, Edge, and Similar Use CasesQuantizationOptimizing Your TensorFlow Model with TF LiteOptimization OptionsPruningKnowledge DistillationTeacher and Student NetworksKnowledge Distillation TechniquesTMKD: Distilling Knowledge for a Q&A TaskIncreasing Robustness by Distilling EfficientNetsConclusion
7. High-Performance Modeling
Distributed TrainingData ParallelismEfficient Input PipelinesInput Pipeline BasicsInput Pipeline Patterns: Improving EfficiencyOptimizing Your Input Pipeline with TensorFlow DataTraining Large Models: The Rise of Giant Neural Nets and ParallelismPotential Solutions and Their ShortcomingsPipeline Parallelism to the Rescue?Conclusion
8. Model Analysis
Analyzing Model PerformanceBlack-Box EvaluationPerformance Metrics and Optimization ObjectivesAdvanced Model AnalysisTensorFlow Model AnalysisThe Learning Interpretability ToolAdvanced Model DebuggingBenchmark ModelsSensitivity AnalysisResidual AnalysisModel RemediationDiscrimination RemediationFairnessFairness EvaluationFairness ConsiderationsContinuous Evaluation and MonitoringConclusion

9. Interpretability
Explainable AIModel Interpretation MethodsMethod CategoriesIntrinsically Interpretable ModelsModel-Agnostic MethodsLocal Interpretable Model-Agnostic Explanations Shapley ValuesThe SHAP Library Testing Concept Activation Vectors AI ExplanationsExample: Exploring Model Sensitivity with SHAPRegression ModelsNatural Language Processing ModelsConclusion
10. Neural Architecture Search
Hyperparameter TuningIntroduction to AutoMLKey Components of NASSearch SpacesSearch StrategiesPerformance Estimation StrategiesAutoML in the CloudAmazon SageMaker AutopilotMicrosoft Azure Automated Machine LearningGoogle Cloud AutoMLUsing AutoMLGenerative AI and AutoMLConclusion
11. Introduction to Model Serving
Model TrainingModel PredictionLatencyThroughputCostResources and Requirements for Serving ModelsCost and ComplexityAcceleratorsFeeding the BeastModel DeploymentsData Center DeploymentsMobile and Distributed DeploymentsModel ServersManaged ServicesConclusion
12. Model Serving Patterns
Batch InferenceBatch ThroughputBatch Inference Use CasesETL for Distributed Batch and Stream Processing SystemsIntroduction to Real-Time InferenceSynchronous Delivery of Real-Time PredictionsAsynchronous Delivery of Real-Time PredictionsOptimizing Real-Time InferenceReal-Time Inference Use CasesServing Model EnsemblesEnsemble TopologiesExample EnsembleEnsemble Serving ConsiderationsModel Routers: Ensembles in GenAIData Preprocessing and Postprocessing in Real TimeTraining Transformations Versus Serving TransformationsWindowingOptions for PreprocessingEnter TensorFlow TransformPostprocessingInference at the Edge and at the BrowserChallengesModel Deployments via ContainersTraining on the DeviceFederated LearningRuntime InteroperabilityInference in Web BrowsersConclusion
13. Model Serving Infrastructure
Model ServersTensorFlow ServingNVIDIA Triton Inference ServerTorchServeBuilding Scalable InfrastructureContainerizationTraditional Deployment EraVirtualized Deployment EraContainer Deployment EraThe Docker Containerization FrameworkContainer OrchestrationReliability and Availability Through RedundancyObservabilityHigh AvailabilityAutomated DeploymentsHardware AcceleratorsGPUsTPUsConclusion
14. Model Serving Examples
Example: Deploying TensorFlow Models with TensorFlow ServingExporting Keras Models for TF ServingSetting Up TF Serving with DockerBasic Configuration of TF ServingMaking Model Prediction Requests with RESTMaking Model Prediction Requests with gRPCGetting Predictions from Classification and Regression ModelsUsing PayloadsGetting Model Metadata from TF ServingMaking Batch Inference RequestsExample: Profiling TF Serving Inferences with TF ProfilerPrerequisitesTensorBoard SetupModel ProfileExample: Basic TorchServe SetupInstalling the TorchServe DependenciesExporting Your Model for TorchServeSetting Up TorchServeMaking Model Prediction RequestsMaking Batch Inference RequestsConclusion
15. Model Management and Delivery
Experiment TrackingExperimenting in NotebooksExperimenting OverallTools for Experiment Tracking and VersioningIntroduction to MLOpsData Scientists Versus Software EngineersML EngineersML in Products and ServicesMLOpsMLOps MethodologyMLOps Level 0MLOps Level 1MLOps Level 2Components of an Orchestrated WorkflowThree Types of Custom ComponentsPython Function–Based ComponentsContainer-Based ComponentsFully Custom ComponentsTFX Deep DiveTFX SDKIntermediate RepresentationRuntimeImplementing an ML Pipeline Using TFX ComponentsAdvanced Features of TFXManaging Model VersionsApproaches to Versioning ModelsModel LineageModel RegistriesContinuous Integration and Continuous DeploymentContinuous IntegrationContinuous DeliveryProgressive DeliveryBlue/Green DeploymentCanary DeploymentLive ExperimentationConclusion
16. Model Monitoring and Logging
The Importance of MonitoringObservability in Machine LearningWhat Should You Monitor?Custom Alerting in TFXLoggingDistributed TracingMonitoring for Model DecayData Drift and Concept DriftModel Decay DetectionSupervised Monitoring TechniquesUnsupervised Monitoring TechniquesMitigating Model DecayRetraining Your ModelWhen to RetrainAutomated RetrainingConclusion
17. Privacy and Legal Requirements
Why Is Data Privacy Important?What Data Needs to Be Kept Private?HarmsOnly Collect What You NeedGenAI Data Scraped from the Web and Other SourcesLegal RequirementsThe GDPR and the CCPAThe GDPR’s Right to Be Forgotten Pseudonymization and AnonymizationDifferential PrivacyLocal and Global DPEpsilon-Delta DPApplying Differential Privacy to MLTensorFlow Privacy ExampleFederated LearningEncrypted MLConclusion
18. Orchestrating Machine Learning Pipelines
An Introduction to Pipeline OrchestrationWhy Pipeline Orchestration?Directed Acyclic GraphsPipeline Orchestration with TFXInteractive TFX PipelinesConverting Your Interactive Pipeline for Production Orchestrating TFX Pipelines with Apache BeamOrchestrating TFX Pipelines with Kubeflow PipelinesIntroduction to Kubeflow PipelinesInstallation and Initial SetupAccessing Kubeflow PipelinesThe Workflow from TFX to KubeflowOpFunc FunctionsOrchestrating Kubeflow PipelinesGoogle Cloud Vertex PipelinesSetting Up Google Cloud and Vertex PipelinesSetting Up a Google Cloud Service AccountOrchestrating Pipelines with Vertex PipelinesExecuting Vertex PipelinesChoosing Your OrchestratorInteractive TFXApache BeamKubeflow PipelinesGoogle Cloud Vertex PipelinesAlternatives to TFXConclusion
19. Advanced TFX
Advanced Pipeline PracticesConfigure Your ComponentsImport ArtifactsUse Resolver Node Execute a Conditional Pipeline Export TF Lite ModelsWarm-Starting Model TrainingUse Exit HandlersTrigger Messages from TFXCustom TFX Components: Architecture and Use CasesArchitecture of TFX ComponentsUse Cases of Custom ComponentsUsing Function-Based Custom ComponentsWriting a Custom Component from ScratchDefining Component SpecificationsDefining Component ChannelsWriting the Custom ExecutorWriting the Custom DriverAssembling the Custom ComponentUsing Our Basic Custom ComponentImplementation ReviewReusing Existing ComponentsCreating Container-Based Custom ComponentsWhich Custom Component Is Right for You?TFX-AddonsConclusion
20. ML Pipelines for Computer Vision Problems
Our DataOur ModelCustom Ingestion ComponentData PreprocessingExporting the Model Our PipelineData IngestionData PreprocessingModel TrainingModel Evaluation Model ExportPutting It All Together Executing on Apache BeamExecuting on Vertex Pipelines Model Deployment with TensorFlow ServingConclusion
21. ML Pipelines for Natural Language Processing
Our DataOur ModelIngestion ComponentData PreprocessingPutting the Pipeline TogetherExecuting the Pipeline Model Deployment with Google Cloud VertexRegistering Your ML ModelCreating a New Model EndpointDeploying Your ML ModelRequesting Predictions from the Deployed ModelCleaning Up Your Deployed ModelConclusion
22. Generative AI
Generative ModelsGenAI Model TypesAgents and CopilotsPretrainingPretraining DatasetsEmbeddingsSelf-Supervised Training with MasksFine-TuningFine-Tuning Versus Transfer LearningFine-Tuning DatasetsFine-Tuning Considerations for ProductionFine-Tuning Versus Model APIsParameter-Efficient Fine-TuningLoRAS-LoRAHuman AlignmentReinforcement Learning from Human FeedbackReinforcement Learning from AI FeedbackDirect Preference OptimizationPromptingChainingRetrieval Augmented GenerationReActEvaluationEvaluation TechniquesBenchmarking Across ModelsLMOpsGenAI AttacksJailbreaksPrompt InjectionResponsible GenAIDesign for ResponsibilityConduct Adversarial TestingConstitutional AIConclusion
23. The Future of Machine Learning Production Systems and Next Steps
Let’s Think in Terms of ML Systems, Not ML ModelsBringing ML Systems Closer to Domain ExpertsPrivacy Has Never Been More ImportantConclusion
Index
About the Authors

Content preview from Machine Learning Production Systems

Chapter 13. Model Serving Infrastructure

Just like any other application, your ML infrastructure can be trained and deployed on premises on your own hardware infrastructure. However, this approach necessitates procurement of the hardware (physical machines) and the GPUs for training and inference of large models (deep neural networks, or DNNs). This can be viable for large companies that run and maintain ML applications for a long time.

The viable option for small to medium-size businesses and individual teams is to deploy on a cloud and leverage the hardware infrastructure provided by cloud service providers such as Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. Most of the popular cloud service providers have specialized training and deployment solutions for ML models. These include AutoML on GCP and Amazon SageMaker Autopilot on AWS.

When you’re deploying ML models on premises (on your own hardware infrastructure), you can use an open source prebuilt model server such as TensorFlow Serving, KServe, or NVIDIA Triton.

If you choose to deploy ML models on a cloud, you can deploy trained models on virtual machines (VMs) such as EC2 or Google Compute Engine, and use model servers such as TensorFlow Serving to serve inference requests. Or you may choose to use compute cluster offerings such as Google Kubernetes Engine.

Cloud service providers also offer solutions for managing the entire ML workflow, including data cleaning, data preparation, feature ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098156008Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Machine Learning Production Systems

by Robert Crowe, Hannes Hapke, Emily Caveness, Di Zhu

Chapter 13. Model Serving Infrastructure

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.