book

Machine Learning Engineering in Action

Name: Machine Learning Engineering in Action
Author: Ben Wilson
ISBN: 9781617298714

by Ben Wilson

April 2022

Intermediate to advanced

576 pages

18h 11m

English

Manning Publications

Read now

Unlock full access

inside front cover
Machine Learning Engineering in Action
Copyright
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
Part 1 An introduction to machine learning engineering
1 What is a machine learning engineer?
1.1 Why ML engineering?1.2 The core tenets of ML engineering1.2.1 Planning1.2.2 Scoping and research1.2.3 Experimentation1.2.4 Development1.2.5 Deployment1.2.6 Evaluation1.3 The goals of ML engineeringSummary
2 Your data science could use some engineering
2.1 Augmenting a complex profession with processes to increase project success2.2 A foundation of simplicity2.3 Co-opting principles of Agile software engineering2.3.1 Communication and cooperation2.3.2 Embracing and expecting change2.4 The foundation of ML engineeringSummary
3 Before you model: Planning and scoping a project
3.1 Planning: You want me to predict what?!3.1.1 Basic planning for a project3.1.2 That first meeting3.1.3 Plan for demos—lots of demos3.1.4 Experimentation by solution building: Wasting time for pride’s sake3.2 Experimental scoping: Setting expectations and boundaries3.2.1 What is experimental scoping?3.2.2 Experimental scoping for the ML team: Research3.2.3 Experimental scoping for the ML team: ExperimentationSummary
4 Before you model: Communication and logistics of projects
4.1 Communication: Defining the problem4.1.1 Understanding the problem4.1.2 Setting critical discussion boundaries4.2 Don’t waste our time: Meeting with cross-functional teams4.2.1 Experimental update meeting: Do we know what we’re doing here?4.2.2 SME review/prototype review: Can we solve this?4.2.3 Development progress review(s): Is this thing going to work?4.2.4 MVP review: Did you build what we asked for?4.2.5 Preproduction review: We really hope we didn’t screw this up4.3 Setting limits on your experimentation4.3.1 Set a time limit4.3.2 Can you put this into production? Would you want to maintain it?4.3.3 TDD vs. RDD vs. PDD vs. CDD for ML projects4.4 Planning for business rules chaos4.4.1 Embracing chaos by planning for it4.4.2 Human-in-the-loop design4.4.3 What’s your backup plan?4.5 Talking about resultsSummary

5 Experimentation in action: Planning and researching an ML project
5.1 Planning experiments5.1.1 Perform basic research and planning5.1.2 Forget the blogs—read the API docs5.1.3 Draw straws for an internal hackathon5.1.4 Level the playing field5.2 Performing experimental prep work5.2.1 Performing data analysis5.2.2 Moving from script to reusable code5.2.3 One last note on building reusable code for experimentationSummary
6 Experimentation in action: Testing and evaluating a project
6.1 Testing ideas6.1.1 Setting guidelines in code6.1.2 Running quick forecasting tests6.2 Whittling down the possibilities6.2.1 Evaluating prototypes properly6.2.2 Making a call on the direction to go in6.2.3 So . . . what’s next?Summary
7 Experimentation in action: Moving from prototype to MVP
7.1 Tuning: Automating the annoying stuff7.1.1 Tuning options7.1.2 Hyperopt primer7.1.3 Using Hyperopt to tune a complex forecasting problem7.2 Choosing the right tech for the platform and the team7.2.1 Why Spark?7.2.2 Handling tuning from the driver with SparkTrials7.2.3 Handling tuning from the workers with a pandas_udf7.2.4 Using new paradigms for teams: Platforms and technologiesSummary
8 Experimentation in action: Finalizing an MVP with MLflow and runtime optimization
8.1 Logging: Code, metrics, and results8.1.1 MLflow tracking8.1.2 Please stop printing and log your information8.1.3 Version control, branch strategies, and working with others8.2 Scalability and concurrency8.2.1 What is concurrency?8.2.2 What you can (and can’t) run asynchronouslySummary
Part 2 Preparing for production: Creating maintainable ML
9 Modularity for ML: Writing testable and legible code
9.1 Understanding monolithic scripts and why they are bad9.1.1 How monoliths come into being9.1.2 Walls of text9.1.3 Considerations for monolithic scripts9.2 Debugging walls of text9.3 Designing modular ML code9.4 Using test-driven development for MLSummary
10 Standards of coding and creating maintainable ML code
10.1 ML code smells10.2 Naming, structure, and code architecture10.2.1 Naming conventions and structure10.2.2 Trying to be too clever10.2.3 Code architecture10.3 Tuple unpacking and maintainable alternatives10.3.1 Tuple unpacking example10.3.2 A solid alternative to tuple unpacking10.4 Blind to issues: Eating exceptions and other bad practices10.4.1 Try/catch with the precision of a shotgun10.4.2 Exception handling with laser precision10.4.3 Handling errors the right way10.5 Use of global mutable objects10.5.1 How mutability can burn you10.5.2 Encapsulation to prevent mutable side effects10.6 Excessively nested logicSummary
11 Model measurement and why it’s so important
11.1 Measuring model attribution11.1.1 Measuring prediction performance11.1.2 Clarifying correlation vs. causation11.2 Leveraging A/B testing for attribution calculations11.2.1 A/B testing 10111.2.2 Evaluating continuous metrics11.2.3 Using alternative displays and tests11.2.4 Evaluating categorical metricsSummary
12 Holding on to your gains by watching for drift
12.1 Detecting drift12.1.1 What influences drift?12.2 Responding to drift12.2.1 What can we do about it?12.2.2 Responding to driftSummary
13 ML development hubris
13.1 Elegant complexity vs. overengineering13.1.1 Lightweight scripted style (imperative)13.1.2 An overengineered mess13.2 Unintentional obfuscation: Could you read this if you didn’t write it?13.2.1 The flavors of obfuscation13.2.2 Troublesome coding habits recap13.3 Premature generalization, premature optimization, and other bad ways to show how smart you are13.3.1 Generalization and frameworks: Avoid them until you can’t13.3.2 Optimizing too early13.4 Do you really want to be the canary? Alpha testing and the dangers of the open source coal mine13.5 Technology-driven development vs. solution-driven developmentSummary
Part 3 Developing production machine learning code
14 Writing production code
14.1 Have you met your data?14.1.1 Make sure you have the data14.1.2 Check your data provenance14.1.3 Find a source of truth and align on it14.1.4 Don’t embed data cleansing into your production code14.2 Monitoring your features14.3 Monitoring everything else in the model life cycle14.4 Keeping things as simple as possible14.4.1 Simplicity in problem definitions14.4.2 Simplicity in implementation14.5 Wireframing ML projects14.6 Avoiding cargo cult ML behaviorSummary
15 Quality and acceptance testing
15.1 Data consistency15.1.1 Training and inference skew15.1.2 A brief intro to feature stores15.1.3 Process over technology15.1.4 The dangers of a data silo15.2 Fallbacks and cold starts15.2.1 Leaning heavily on prior art15.2.2 Cold-start woes15.3 End user vs. internal use testing15.3.1 Biased testing15.3.2 Dogfooding15.3.3 SME evaluation15.4 Model interpretability15.4.1 Shapley additive explanations15.4.2 Using shapSummary
16 Production infrastructure
16.1 Artifact management16.1.1 MLflow’s model registry16.1.2 Interfacing with the model registry16.2 Feature stores16.2.1 What a feature store is used for16.2.2 Using a feature store16.2.3 Evaluating a feature store16.3 Prediction serving architecture16.3.1 Determining serving needs16.3.2 Bulk external delivery16.3.3 Microbatch streaming16.3.4 Real-time server-side16.3.5 Integrated models (edge deployment)Summary
Appendix A. Big O(no) and how to think about runtime performance
A.1 What is Big O, anyway?A.1.1 A gentle introduction to complexityA.2 Complexity by exampleA.2.1 O(1): The “It doesn’t matter how big the data is” algorithmA.2.2 O(n): The linear relationship algorithmA.2.3 O(n2): A polynomial relationship to the size of the collectionA.3 Analyzing decision-tree complexityA.4 General algorithmic complexity for ML
Appendix B. Setting up a development environment
B.1 The case for a clean experimentation environmentB.2 Containers to deal with dependency hellB.3 Creating a container-based pristine environment for experimentation
index
inside back cover

Content preview from Machine Learning Engineering in Action

11 Model measurement and why it’s so important

This chapter covers

Methodologies for determining the impact of a model
A/B testing approaches for attribution data collection

In part 1, we focused on aligning ML project work to business problems. This is, after all, the most critical aspect for making the solution viable. While those earlier chapters focused on communication before, during, and immediately up to a production release, this chapter focuses on the project communication after release. We’ll cover how to present, discuss, and accurately report on the long-term health of ML projects—specifically, in language and methodologies that the business will understand.

Discussions about model performance are complex. While the business is ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Feature Engineering for Machine Learning

Publisher Resources

ISBN: 9781617298714Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Machine Learning Engineering in Action

by Ben Wilson

11 Model measurement and why it’s so important

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.