Introducing MLOps

Table of contents

1. Preface
   1. Who This Book Is For
   2. How This Book Is Organized
   3. Conventions Used in This Book
   4. O’Reilly Online Learning
   5. How to Contact Us
   6. Acknowledgments
2. I. MLOps: What and Why
3. 1. Why Now and Challenges
   1. Defining MLOps and Its Challenges
   2. MLOps to Mitigate Risk
      1. Risk Assessment
      2. Risk Mitigation
      3. MLOps for Responsible AI
   3. MLOps for Scale
   4. Closing Thoughts
4. 2. People of MLOps
   1. Subject Matter Experts
   2. Data Scientists
   3. Data Engineers
   4. Software Engineers
   5. DevOps
   6. Model Risk Manager/Auditor
   7. Machine Learning Architect
   8. Closing Thoughts
5. 3. Key MLOps Features
   1. A Primer on Machine Learning
   2. Model Development
      1. Establishing Business Objectives
      2. Data Sources and Exploratory Data Analysis
      3. Feature Engineering and Selection
      4. Training and Evaluation
      5. Reproducibility
      6. Responsible AI
   3. Productionalization and Deployment
      1. Model Deployment Types and Contents
      2. Model Deployment Requirements
   4. Monitoring
      1. DevOps Concerns
      2. Data Scientist Concerns
      3. Business Concerns
   5. Iteration and Life Cycle
      1. Iteration
      2. The Feedback Loop
   6. Governance
      1. Data Governance
      2. Process Governance
   7. Closing Thoughts
6. II. MLOps: How
7. 4. Developing Models
   1. What Is a Machine Learning Model?
      1. In Theory
      2. In Practice
      3. Required Components
      4. Different ML Algorithms, Different MLOps Challenges
   2. Data Exploration
   3. Feature Engineering and Selection
      1. Feature Engineering Techniques
      2. How Feature Selection Impacts MLOps Strategy
   4. Experimentation
   5. Evaluating and Comparing Models
      1. Choosing Evaluation Metrics
      2. Cross-Checking Model Behavior
      3. Impact of Responsible AI on Modeling
   6. Version Management and Reproducibility
   7. Closing Thoughts
8. 5. Preparing for Production
   1. Runtime Environments
      1. Adaptation from Development to Production Environments
      2. Data Access Before Validation and Launch to Production
      3. Final Thoughts on Runtime Environments
   2. Model Risk Evaluation
      1. The Purpose of Model Validation
      2. The Origins of ML Model Risk
   3. Quality Assurance for Machine Learning
   4. Key Testing Considerations
   5. Reproducibility and Auditability
   6. Machine Learning Security
      1. Adversarial Attacks
      2. Other Vulnerabilities
   7. Model Risk Mitigation
      1. Changing Environments
      2. Interactions Between Models
      3. Model Misbehavior
   8. Closing Thoughts
9. 6. Deploying to Production
   1. CI/CD Pipelines
   2. Building ML Artifacts
      1. What’s in an ML Artifact?
      2. The Testing Pipeline
   3. Deployment Strategies
      1. Categories of Model Deployment
      2. Considerations When Sending Models to Production
      3. Maintenance in Production
   4. Containerization
   5. Scaling Deployments
   6. Requirements and Challenges
   7. Closing Thoughts
10. 7. Monitoring and Feedback Loop
    1. How Often Should Models Be Retrained?
    2. Understanding Model Degradation
       1. Ground Truth Evaluation
       2. Input Drift Detection
    3. Drift Detection in Practice
       1. Example Causes of Data Drift
       2. Input Drift Detection Techniques
    4. The Feedback Loop
       1. Logging
       2. Model Evaluation
       3. Online Evaluation
    5. Closing Thoughts
11. 8. Model Governance
    1. Who Decides What Governance the Organization Needs?
    2. Matching Governance with Risk Level
    3. Current Regulations Driving MLOps Governance
       1. Pharmaceutical Regulation in the US: GxP
       2. Financial Model Risk Management Regulation
       3. GDPR and CCPA Data Privacy Regulations
    4. The New Wave of AI-Specific Regulations
    5. The Emergence of Responsible AI
    6. Key Elements of Responsible AI
       1. Element 1: Data
       2. Element 2: Bias
       3. Element 3: Inclusiveness
       4. Element 4: Model Management at Scale
       5. Element 5: Governance
    7. A Template for MLOps Governance
       1. Step 1: Understand and Classify the Analytics Use Cases
       2. Step 2: Establish an Ethical Position
       3. Step 3: Establish Responsibilities
       4. Step 4: Determine Governance Policies
       5. Step 5: Integrate Policies into the MLOps Process
       6. Step 6: Select the Tools for Centralized Governance Management
       7. Step 7: Engage and Educate
       8. Step 8: Monitor and Refine
    8. Closing Thoughts
12. III. MLOps: Real-World Examples
13. 9. MLOps in Practice: Consumer Credit Risk Management
    1. Background: The Business Use Case
    2. Model Development
    3. Model Bias Considerations
    4. Prepare for Production
    5. Deploy to Production
    6. Closing Thoughts
14. 10. MLOps in Practice: Marketing Recommendation Engines
    1. The Rise of Recommendation Engines
       1. The Role of Machine Learning
       2. Push or Pull?
    2. Data Preparation
    3. Design and Manage Experiments
    4. Model Training and Deployment
       1. Scalability and Customizability
       2. Monitoring and Retraining Strategy
       3. Real-Time Scoring
       4. Ability to Turn Recommendations On and Off
    5. Pipeline Structure and Deployment Strategy
    6. Monitoring and Feedback
       1. Retraining Models
       2. Updating Models
       3. Runs Overnight, Sleeps During Daytime
       4. Option to Manually Control Models
       5. Option to Automatically Control Models
       6. Monitoring Performance
    7. Closing Thoughts
15. 11. MLOps in Practice: Consumption Forecast
    1. Power Systems
    2. Data Collection
    3. Problem Definition: Machine Learning, or Not Machine Learning?
    4. Spatial and Temporal Resolution
    5. Implementation
    6. Modeling
    7. Deployment
    8. Monitoring
    9. Closing Thoughts
16. Index