book

Effective Machine Learning Teams

by David Tan, Ada Leung, David Colls

February 2024

Beginner to intermediate

402 pages

11h 33m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Includes

Includes Quizzes

Who This Book Is ForHow This Book Is OrganizedPart I: Product and DeliveryPart II: EngineeringPart III: TeamsAdditional ThoughtsConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgmentsFrom David TanFrom Ada LeungFrom David Colls
ML: Promises and DisappointmentsContinued Optimism in MLWhy ML Projects FailIs There a Better Way? How Systems Thinking and Lean Can HelpYou Can’t “MLOps” Your Problems AwaySee the Whole: A Systems Thinking Lens for Effective ML DeliveryThe Five Disciplines Required for Effective ML DeliveryConclusion
ML Product DiscoveryDiscovering Product OpportunitiesCanvases to Define Product OpportunitiesTechniques for Rapidly Designing, Delivering, and Testing SolutionsInception: Setting Teams Up for SuccessInception: What Is It and How Do We Do It?How to Plan and Run an InceptionUser Stories: Building Blocks of an MVPProduct DeliveryCadence of Delivery ActivitiesMeasuring Product and DeliveryConclusion
What If Our Code Worked Everywhere, Every Time?A Better Way: Check Out and GoPrinciples for Effective Dependency ManagementTools for Dependency ManagementA Crash Course on Docker and batectWhat Are Containers?Reduce the Number of Moving Parts in Docker with batectConclusion
In Context: ML Development WorkflowIdentifying What to ContainerizeHands-On Exercise: Reproducible Development Environments, Aided by ContainersSecure Dependency ManagementRemove Unnecessary DependenciesAutomate Checks for Security VulnerabilitiesConclusion
Automated Tests: The Foundation for Iterating Quickly and ReliablyStarting with Why: Benefits of Test AutomationIf Automated Testing Is So Important, Why Aren’t We Doing It?Building Blocks for a Comprehensive Test Strategy for ML SystemsThe What: Identifying Components For TestingCharacteristics of a Good Test and Pitfalls to AvoidThe How: Structure of a TestSoftware TestsUnit TestsTraining Smoke TestsAPI TestsPost-deployment TestsConclusion
Model TestsThe Necessity of Model TestsChallenges of Testing ML ModelsFitness Functions for ML ModelsModel Metrics Tests (Global and Stratified)Behavioral TestsTesting Large Language Models: Why and HowEssential Complementary Practices for Model TestsError Analysis and VisualizationLearn from Production by Closing the Data Collection LoopOpen-Closed Test DesignExploratory TestingMeans to Improve the ModelDesigning to Minimize the Cost of FailuresMonitoring in ProductionBringing It All TogetherNext Steps: Applying What You’ve LearnedMake Incremental ImprovementsDemonstrate ValueConclusion
The Benefits (and Surprising Simplicity) of Knowing Our IDEWhy Should We Care About IDEs?If IDEs Are So Important, Why Haven’t I Learned About Them Yet?The Plan: Getting Productive in Two StagesStage 1: Configuring Your IDEStage 2: The Star of the Show—Keyboard ShortcutsYou Did It!Conclusion

Technical Debt: The Sand in Our GearsGetting to a Healthy Level of Debt Through Tests, Design, and RefactoringRefactoring 101How to Refactor a Notebook (or a Problematic Codebase)The Map: Planning Your JourneyThe Journey: Hitting the RoadLooking Back at What We’ve AchievedTechnical Debt Management in the Real WorldTechnical Debt Management TechniquesA Positive Lens on Debt: Systems Health RatingsConclusion: Make Good Easy
MLOps: Strengths and Missing Puzzle PiecesMLOps 101Smells: Hints That We Missed SomethingContinuous Delivery for Machine LearningBenefits of CD4MLA Crash Course on Continuous Delivery PrinciplesBuilding Blocks of CD4ML: Creating a Production-Ready ML SystemHow CD4ML Supports ML Governance and Responsible AIConclusion
Common Challenges Faced by ML TeamsEffective Team InternalsTrust as the Foundational Building BlockCommunicationDiverse MembershipPurposeful, Shared ProgressInternal Tactics to Build Effective TeamsImproving Flow with Engineering EffectivenessFeedback LoopsCognitive LoadFlow StateConclusion
Common Challenges Faced by ML OrganizationsEffective Organizations as Teams of TeamsThe Role of Value-Driven Portfolio ManagementTeam Topologies ModelTeam Topologies for ML TeamsOrganizational Tactics to Build Effective TeamsIntentional LeadershipCreate Structures and Systems for Effective TeamsEngage Stakeholders and Coordinate Organizational ResourcesCultivate Psychological SafetyChampion Continuous ImprovementEmbrace Failure as a Learning OpportunityBuild the Culture We Wish We HadEncourage Teams to Play at WorkConclusionEpilogue: Dana’s Journey

Content preview from Effective Machine Learning Teams

Chapter 5. Automated Testing: Move Fast Without Breaking Things

Dana has just sat down at her desk, the aroma of fresh coffee filling the space around her. She and a junior data scientist on the team continued on the user story that they kicked off yesterday—engineering a new feature that could improve the model.

They made the necessary changes and executed a command to run the tests. This suite of tests helped validate that the entire codebase was still behaving as expected. After 20 seconds, their terminal showed dashes of green—all the tests passed.

Sometimes the terminal went red. Some tests failed, but that’s OK—the failing tests caught them as they were about to fall into a deep rabbit hole, helping them recover easily by tracing a few steps backward. The tests are now back to green and they gave it another go.

Green or red, the tests gave them fast feedback on code changes. The tests gave them confidence and the occasional dopamine hit to tell them if they were going in the right direction or stop them when they went in the wrong direction. They didn’t have to follow a tedious sequence of manual steps to test the code. When the tests failed, there were only a small number of changes that could have caused the failure, not hours of potential suspects to sort through.

When they needed to train the model, they ran a command that triggered training on the cloud and their experiment-tracking dashboard lit up with updated metrics and explainability visualizations, which signaled ...