book

Effective Machine Learning Teams

by David Tan, Ada Leung, David Colls

February 2024

Beginner to intermediate

402 pages

11h 33m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Who This Book Is ForHow This Book Is OrganizedPart I: Product and DeliveryPart II: EngineeringPart III: TeamsAdditional ThoughtsConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsFrom David TanFrom Ada LeungFrom David Colls
ML: Promises and DisappointmentsContinued Optimism in MLWhy ML Projects FailIs There a Better Way? How Systems Thinking and Lean Can HelpYou Can’t “MLOps” Your Problems AwaySee the Whole: A Systems Thinking Lens for Effective ML DeliveryThe Five Disciplines Required for Effective ML DeliveryConclusion
ML Product DiscoveryDiscovering Product OpportunitiesCanvases to Define Product OpportunitiesTechniques for Rapidly Designing, Delivering, and Testing SolutionsInception: Setting Teams Up for SuccessInception: What Is It and How Do We Do It?How to Plan and Run an InceptionUser Stories: Building Blocks of an MVPProduct DeliveryCadence of Delivery ActivitiesMeasuring Product and DeliveryConclusion
What If Our Code Worked Everywhere, Every Time?A Better Way: Check Out and GoPrinciples for Effective Dependency ManagementTools for Dependency ManagementA Crash Course on Docker and batectWhat Are Containers?Reduce the Number of Moving Parts in Docker with batectConclusion
In Context: ML Development WorkflowIdentifying What to ContainerizeHands-On Exercise: Reproducible Development Environments, Aided by ContainersSecure Dependency ManagementRemove Unnecessary DependenciesAutomate Checks for Security VulnerabilitiesConclusion
Automated Tests: The Foundation for Iterating Quickly and ReliablyStarting with Why: Benefits of Test AutomationIf Automated Testing Is So Important, Why Aren’t We Doing It?Building Blocks for a Comprehensive Test Strategy for ML SystemsThe What: Identifying Components For TestingCharacteristics of a Good Test and Pitfalls to AvoidThe How: Structure of a TestSoftware TestsUnit TestsTraining Smoke TestsAPI TestsPost-deployment TestsConclusion
Model TestsThe Necessity of Model TestsChallenges of Testing ML ModelsFitness Functions for ML ModelsModel Metrics Tests (Global and Stratified)Behavioral TestsTesting Large Language Models: Why and HowEssential Complementary Practices for Model TestsError Analysis and VisualizationLearn from Production by Closing the Data Collection LoopOpen-Closed Test DesignExploratory TestingMeans to Improve the ModelDesigning to Minimize the Cost of FailuresMonitoring in ProductionBringing It All TogetherNext Steps: Applying What You’ve LearnedMake Incremental ImprovementsDemonstrate ValueConclusion
The Benefits (and Surprising Simplicity) of Knowing Our IDEWhy Should We Care About IDEs?If IDEs Are So Important, Why Haven’t I Learned About Them Yet?The Plan: Getting Productive in Two StagesStage 1: Configuring Your IDEStage 2: The Star of the Show—Keyboard ShortcutsYou Did It!Conclusion

Technical Debt: The Sand in Our GearsGetting to a Healthy Level of Debt Through Tests, Design, and RefactoringRefactoring 101How to Refactor a Notebook (or a Problematic Codebase)The Map: Planning Your JourneyThe Journey: Hitting the RoadLooking Back at What We’ve AchievedTechnical Debt Management in the Real WorldTechnical Debt Management TechniquesA Positive Lens on Debt: Systems Health RatingsConclusion: Make Good Easy
MLOps: Strengths and Missing Puzzle PiecesMLOps 101Smells: Hints That We Missed SomethingContinuous Delivery for Machine LearningBenefits of CD4MLA Crash Course on Continuous Delivery PrinciplesBuilding Blocks of CD4ML: Creating a Production-Ready ML SystemHow CD4ML Supports ML Governance and Responsible AIConclusion
Common Challenges Faced by ML TeamsEffective Team InternalsTrust as the Foundational Building BlockCommunicationDiverse MembershipPurposeful, Shared ProgressInternal Tactics to Build Effective TeamsImproving Flow with Engineering EffectivenessFeedback LoopsCognitive LoadFlow StateConclusion
Common Challenges Faced by ML OrganizationsEffective Organizations as Teams of TeamsThe Role of Value-Driven Portfolio ManagementTeam Topologies ModelTeam Topologies for ML TeamsOrganizational Tactics to Build Effective TeamsIntentional LeadershipCreate Structures and Systems for Effective TeamsEngage Stakeholders and Coordinate Organizational ResourcesCultivate Psychological SafetyChampion Continuous ImprovementEmbrace Failure as a Learning OpportunityBuild the Culture We Wish We HadEncourage Teams to Play at WorkConclusionEpilogue: Dana’s Journey

Content preview from Effective Machine Learning Teams

Chapter 9. MLOps and Continuous Delivery for ML (CD4ML)

That anxiety makes its appearance is the pivot upon which everything turns.

Søren Kierkegaard, The Concept of Anxiety

It’s 10:36 a.m. Dana is pairing with Ted, an infrastructure engineer, to deploy the new model her team has been working on for several months. The energy in the room is mixed with determination and anxiety—it’s a new model for a high-profile release. They’ve been testing the model for three weeks, but the next hurdle—deployment to production—has typically been fraught with issues and numerous retries.

As they navigate the labyrinthine web of deployment scripts, configuration files, and infrastructure components, Dana couldn’t help but feel that something was amiss. She wasn’t confident that their test dataset was representative of what the model would be seeing in production. It didn’t help that the complexity of the system was so overwhelming—the sheer number of moving parts made it difficult to get a sense of where things might go wrong.

It’s 12:45 p.m. Dana and Ted completed the last of their deployment procedures 10 minutes ago, but the trickling stream of alerts is a cruel reminder that something has gone awry.

It’s 7:10 p.m. After hours of troubleshooting, a fix has finally been deployed. Dana and Ted exhale relief and finally go home, exhausted.

A week has passed, and one Tuesday morning Dana hears a scrape-triple-knock on her Slack. A message at 8:45 a.m.—it is Sarah from product analytics. She informs ...