book

Machine Learning Engineering in Action

by Ben Wilson

April 2022

Intermediate to advanced

576 pages

18h 11m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1.1 Why ML engineering?1.2 The core tenets of ML engineering1.2.1 Planning1.2.2 Scoping and research1.2.3 Experimentation1.2.4 Development1.2.5 Deployment1.2.6 Evaluation1.3 The goals of ML engineeringSummary
2.1 Augmenting a complex profession with processes to increase project success2.2 A foundation of simplicity2.3 Co-opting principles of Agile software engineering2.3.1 Communication and cooperation2.3.2 Embracing and expecting change2.4 The foundation of ML engineeringSummary
3.1 Planning: You want me to predict what?!3.1.1 Basic planning for a project3.1.2 That first meeting3.1.3 Plan for demos—lots of demos3.1.4 Experimentation by solution building: Wasting time for pride’s sake3.2 Experimental scoping: Setting expectations and boundaries3.2.1 What is experimental scoping?3.2.2 Experimental scoping for the ML team: Research3.2.3 Experimental scoping for the ML team: ExperimentationSummary
4.1 Communication: Defining the problem4.1.1 Understanding the problem4.1.2 Setting critical discussion boundaries4.2 Don’t waste our time: Meeting with cross-functional teams4.2.1 Experimental update meeting: Do we know what we’re doing here?4.2.2 SME review/prototype review: Can we solve this?4.2.3 Development progress review(s): Is this thing going to work?4.2.4 MVP review: Did you build what we asked for?4.2.5 Preproduction review: We really hope we didn’t screw this up4.3 Setting limits on your experimentation4.3.1 Set a time limit4.3.2 Can you put this into production? Would you want to maintain it?4.3.3 TDD vs. RDD vs. PDD vs. CDD for ML projects4.4 Planning for business rules chaos4.4.1 Embracing chaos by planning for it4.4.2 Human-in-the-loop design4.4.3 What’s your backup plan?4.5 Talking about resultsSummary

5.1 Planning experiments5.1.1 Perform basic research and planning5.1.2 Forget the blogs—read the API docs5.1.3 Draw straws for an internal hackathon5.1.4 Level the playing field5.2 Performing experimental prep work5.2.1 Performing data analysis5.2.2 Moving from script to reusable code5.2.3 One last note on building reusable code for experimentationSummary
6.1 Testing ideas6.1.1 Setting guidelines in code6.1.2 Running quick forecasting tests6.2 Whittling down the possibilities6.2.1 Evaluating prototypes properly6.2.2 Making a call on the direction to go in6.2.3 So . . . what’s next?Summary
7.1 Tuning: Automating the annoying stuff7.1.1 Tuning options7.1.2 Hyperopt primer7.1.3 Using Hyperopt to tune a complex forecasting problem7.2 Choosing the right tech for the platform and the team7.2.1 Why Spark?7.2.2 Handling tuning from the driver with SparkTrials7.2.3 Handling tuning from the workers with a pandas_udf7.2.4 Using new paradigms for teams: Platforms and technologiesSummary
8.1 Logging: Code, metrics, and results8.1.1 MLflow tracking8.1.2 Please stop printing and log your information8.1.3 Version control, branch strategies, and working with others8.2 Scalability and concurrency8.2.1 What is concurrency?8.2.2 What you can (and can’t) run asynchronouslySummary
9.1 Understanding monolithic scripts and why they are bad9.1.1 How monoliths come into being9.1.2 Walls of text9.1.3 Considerations for monolithic scripts9.2 Debugging walls of text9.3 Designing modular ML code9.4 Using test-driven development for MLSummary
10.1 ML code smells10.2 Naming, structure, and code architecture10.2.1 Naming conventions and structure10.2.2 Trying to be too clever10.2.3 Code architecture10.3 Tuple unpacking and maintainable alternatives10.3.1 Tuple unpacking example10.3.2 A solid alternative to tuple unpacking10.4 Blind to issues: Eating exceptions and other bad practices10.4.1 Try/catch with the precision of a shotgun10.4.2 Exception handling with laser precision10.4.3 Handling errors the right way10.5 Use of global mutable objects10.5.1 How mutability can burn you10.5.2 Encapsulation to prevent mutable side effects10.6 Excessively nested logicSummary
11.1 Measuring model attribution11.1.1 Measuring prediction performance11.1.2 Clarifying correlation vs. causation11.2 Leveraging A/B testing for attribution calculations11.2.1 A/B testing 10111.2.2 Evaluating continuous metrics11.2.3 Using alternative displays and tests11.2.4 Evaluating categorical metricsSummary
12.1 Detecting drift12.1.1 What influences drift?12.2 Responding to drift12.2.1 What can we do about it?12.2.2 Responding to driftSummary
13.1 Elegant complexity vs. overengineering13.1.1 Lightweight scripted style (imperative)13.1.2 An overengineered mess13.2 Unintentional obfuscation: Could you read this if you didn’t write it?13.2.1 The flavors of obfuscation13.2.2 Troublesome coding habits recap13.3 Premature generalization, premature optimization, and other bad ways to show how smart you are13.3.1 Generalization and frameworks: Avoid them until you can’t13.3.2 Optimizing too early13.4 Do you really want to be the canary? Alpha testing and the dangers of the open source coal mine13.5 Technology-driven development vs. solution-driven developmentSummary
14.1 Have you met your data?14.1.1 Make sure you have the data14.1.2 Check your data provenance14.1.3 Find a source of truth and align on it14.1.4 Don’t embed data cleansing into your production code14.2 Monitoring your features14.3 Monitoring everything else in the model life cycle14.4 Keeping things as simple as possible14.4.1 Simplicity in problem definitions14.4.2 Simplicity in implementation14.5 Wireframing ML projects14.6 Avoiding cargo cult ML behaviorSummary
15.1 Data consistency15.1.1 Training and inference skew15.1.2 A brief intro to feature stores15.1.3 Process over technology15.1.4 The dangers of a data silo15.2 Fallbacks and cold starts15.2.1 Leaning heavily on prior art15.2.2 Cold-start woes15.3 End user vs. internal use testing15.3.1 Biased testing15.3.2 Dogfooding15.3.3 SME evaluation15.4 Model interpretability15.4.1 Shapley additive explanations15.4.2 Using shapSummary
16.1 Artifact management16.1.1 MLflow’s model registry16.1.2 Interfacing with the model registry16.2 Feature stores16.2.1 What a feature store is used for16.2.2 Using a feature store16.2.3 Evaluating a feature store16.3 Prediction serving architecture16.3.1 Determining serving needs16.3.2 Bulk external delivery16.3.3 Microbatch streaming16.3.4 Real-time server-side16.3.5 Integrated models (edge deployment)Summary
A.1 What is Big O, anyway?A.1.1 A gentle introduction to complexityA.2 Complexity by exampleA.2.1 O(1): The “It doesn’t matter how big the data is” algorithmA.2.2 O(n): The linear relationship algorithmA.2.3 O(n2): A polynomial relationship to the size of the collectionA.3 Analyzing decision-tree complexityA.4 General algorithmic complexity for ML
B.1 The case for a clean experimentation environmentB.2 Containers to deal with dependency hellB.3 Creating a container-based pristine environment for experimentation

Content preview from Machine Learning Engineering in Action

11 Model measurement and why it’s so important

This chapter covers

Methodologies for determining the impact of a model
A/B testing approaches for attribution data collection

In part 1, we focused on aligning ML project work to business problems. This is, after all, the most critical aspect for making the solution viable. While those earlier chapters focused on communication before, during, and immediately up to a production release, this chapter focuses on the project communication after release. We’ll cover how to present, discuss, and accurately report on the long-term health of ML projects—specifically, in language and methodologies that the business will understand.

Discussions about model performance are complex. While the business is ...