Effective Machine Learning Teams

Table of contents

1. Preface
   1. Who This Book Is For
   2. How This Book Is Organized
      1. Part I: Product and Delivery
      2. Part II: Engineering
      3. Part III: Teams
   3. Additional Thoughts
   4. Conventions Used in This Book
   5. Using Code Examples
   6. O’Reilly Online Learning
   7. How to Contact Us
   8. Acknowledgments
      1. From David Tan
      2. From Ada Leung
      3. From David Colls
2. 1. Challenges and Better Paths in Delivering ML Solutions
   1. ML: Promises and Disappointments
      1. Continued Optimism in ML
      2. Why ML Projects Fail
   2. Is There a Better Way? How Systems Thinking and Lean Can Help
      1. You Can’t “MLOps” Your Problems Away
      2. See the Whole: A Systems Thinking Lens for Effective ML Delivery
      3. The Five Disciplines Required for Effective ML Delivery
   3. Conclusion
3. I. Product and Delivery
4. 2. Product and Delivery Practices for ML Teams
   1. ML Product Discovery
      1. Discovering Product Opportunities
      2. Canvases to Define Product Opportunities
      3. Techniques for Rapidly Designing, Delivering, and Testing Solutions
   2. Inception: Setting Teams Up for Success
      1. Inception: What Is It and How Do We Do It?
      2. How to Plan and Run an Inception
      3. User Stories: Building Blocks of an MVP
   3. Product Delivery
      1. Cadence of Delivery Activities
      2. Measuring Product and Delivery
   4. Conclusion
5. II. Engineering
6. 3. Effective Dependency Management: Principles and Tools
   1. What If Our Code Worked Everywhere, Every Time?
      1. A Better Way: Check Out and Go
      2. Principles for Effective Dependency Management
      3. Tools for Dependency Management
   2. A Crash Course on Docker and batect
      1. What Are Containers?
      2. Reduce the Number of Moving Parts in Docker with batect
   3. Conclusion
7. 4. Effective Dependency Management in Practice
   1. In Context: ML Development Workflow
      1. Identifying What to Containerize
      2. Hands-On Exercise: Reproducible Development Environments, Aided by Containers
   2. Secure Dependency Management
      1. Remove Unnecessary Dependencies
      2. Automate Checks for Security Vulnerabilities
   3. Conclusion
8. 5. Automated Testing: Move Fast Without Breaking Things
   1. Automated Tests: The Foundation for Iterating Quickly and Reliably
      1. Starting with Why: Benefits of Test Automation
      2. If Automated Testing Is So Important, Why Aren’t We Doing It?
   2. Building Blocks for a Comprehensive Test Strategy for ML Systems
      1. The What: Identifying Components For Testing
      2. Characteristics of a Good Test and Pitfalls to Avoid
      3. The How: Structure of a Test
   3. Software Tests
      1. Unit Tests
      2. Training Smoke Tests
      3. API Tests
      4. Post-deployment Tests
   4. Conclusion
9. 6. Automated Testing: ML Model Tests
   1. Model Tests
      1. The Necessity of Model Tests
      2. Challenges of Testing ML Models
      3. Fitness Functions for ML Models
      4. Model Metrics Tests (Global and Stratified)
      5. Behavioral Tests
      6. Testing Large Language Models: Why and How
   2. Essential Complementary Practices for Model Tests
      1. Error Analysis and Visualization
      2. Learn from Production by Closing the Data Collection Loop
      3. Open-Closed Test Design
      4. Exploratory Testing
      5. Means to Improve the Model
      6. Designing to Minimize the Cost of Failures
      7. Monitoring in Production
      8. Bringing It All Together
   3. Next Steps: Applying What You’ve Learned
      1. Make Incremental Improvements
      2. Demonstrate Value
   4. Conclusion
10. 7. Supercharging Your Code Editor with Simple Techniques
    1. The Benefits (and Surprising Simplicity) of Knowing Our IDE
       1. Why Should We Care About IDEs?
       2. If IDEs Are So Important, Why Haven’t I Learned About Them Yet?
    2. The Plan: Getting Productive in Two Stages
       1. Stage 1: Configuring Your IDE
       2. Stage 2: The Star of the Show—Keyboard Shortcuts
       3. You Did It!
    3. Conclusion
11. 8. Refactoring and Technical Debt Management
    1. Technical Debt: The Sand in Our Gears
       1. Getting to a Healthy Level of Debt Through Tests, Design, and Refactoring
       2. Refactoring 101
    2. How to Refactor a Notebook (or a Problematic Codebase)
       1. The Map: Planning Your Journey
       2. The Journey: Hitting the Road
       3. Looking Back at What We’ve Achieved
    3. Technical Debt Management in the Real World
       1. Technical Debt Management Techniques
       2. A Positive Lens on Debt: Systems Health Ratings
    4. Conclusion: Make Good Easy
12. 9. MLOps and Continuous Delivery for ML (CD4ML)
    1. MLOps: Strengths and Missing Puzzle Pieces
       1. MLOps 101
       2. Smells: Hints That We Missed Something
    2. Continuous Delivery for Machine Learning
       1. Benefits of CD4ML
       2. A Crash Course on Continuous Delivery Principles
       3. Building Blocks of CD4ML: Creating a Production-Ready ML System
    3. How CD4ML Supports ML Governance and Responsible AI
    4. Conclusion
13. III. Teams
14. 10. Building Blocks of Effective ML Teams
    1. Common Challenges Faced by ML Teams
    2. Effective Team Internals
       1. Trust as the Foundational Building Block
       2. Communication
       3. Diverse Membership
       4. Purposeful, Shared Progress
       5. Internal Tactics to Build Effective Teams
    3. Improving Flow with Engineering Effectiveness
       1. Feedback Loops
       2. Cognitive Load
       3. Flow State
    4. Conclusion
15. 11. Effective ML Organizations
    1. Common Challenges Faced by ML Organizations
    2. Effective Organizations as Teams of Teams
       1. The Role of Value-Driven Portfolio Management
       2. Team Topologies Model
       3. Team Topologies for ML Teams
       4. Organizational Tactics to Build Effective Teams
    3. Intentional Leadership
       1. Create Structures and Systems for Effective Teams
       2. Engage Stakeholders and Coordinate Organizational Resources
       3. Cultivate Psychological Safety
       4. Champion Continuous Improvement
       5. Embrace Failure as a Learning Opportunity
       6. Build the Culture We Wish We Had
       7. Encourage Teams to Play at Work
    4. Conclusion
       1. Epilogue: Dana’s Journey
16. Index
17. About the Authors