book

Machine Learning Engineering in Action

by Ben Wilson

April 2022

Intermediate to advanced

576 pages

18h 11m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1.1 Why ML engineering?1.2 The core tenets of ML engineering1.2.1 Planning1.2.2 Scoping and research1.2.3 Experimentation1.2.4 Development1.2.5 Deployment1.2.6 Evaluation1.3 The goals of ML engineeringSummary
2.1 Augmenting a complex profession with processes to increase project success2.2 A foundation of simplicity2.3 Co-opting principles of Agile software engineering2.3.1 Communication and cooperation2.3.2 Embracing and expecting change2.4 The foundation of ML engineeringSummary
3.1 Planning: You want me to predict what?!3.1.1 Basic planning for a project3.1.2 That first meeting3.1.3 Plan for demos—lots of demos3.1.4 Experimentation by solution building: Wasting time for pride’s sake3.2 Experimental scoping: Setting expectations and boundaries3.2.1 What is experimental scoping?3.2.2 Experimental scoping for the ML team: Research3.2.3 Experimental scoping for the ML team: ExperimentationSummary
4.1 Communication: Defining the problem4.1.1 Understanding the problem4.1.2 Setting critical discussion boundaries4.2 Don’t waste our time: Meeting with cross-functional teams4.2.1 Experimental update meeting: Do we know what we’re doing here?4.2.2 SME review/prototype review: Can we solve this?4.2.3 Development progress review(s): Is this thing going to work?4.2.4 MVP review: Did you build what we asked for?4.2.5 Preproduction review: We really hope we didn’t screw this up4.3 Setting limits on your experimentation4.3.1 Set a time limit4.3.2 Can you put this into production? Would you want to maintain it?4.3.3 TDD vs. RDD vs. PDD vs. CDD for ML projects4.4 Planning for business rules chaos4.4.1 Embracing chaos by planning for it4.4.2 Human-in-the-loop design4.4.3 What’s your backup plan?4.5 Talking about resultsSummary

5.1 Planning experiments5.1.1 Perform basic research and planning5.1.2 Forget the blogs—read the API docs5.1.3 Draw straws for an internal hackathon5.1.4 Level the playing field5.2 Performing experimental prep work5.2.1 Performing data analysis5.2.2 Moving from script to reusable code5.2.3 One last note on building reusable code for experimentationSummary
6.1 Testing ideas6.1.1 Setting guidelines in code6.1.2 Running quick forecasting tests6.2 Whittling down the possibilities6.2.1 Evaluating prototypes properly6.2.2 Making a call on the direction to go in6.2.3 So . . . what’s next?Summary
7.1 Tuning: Automating the annoying stuff7.1.1 Tuning options7.1.2 Hyperopt primer7.1.3 Using Hyperopt to tune a complex forecasting problem7.2 Choosing the right tech for the platform and the team7.2.1 Why Spark?7.2.2 Handling tuning from the driver with SparkTrials7.2.3 Handling tuning from the workers with a pandas_udf7.2.4 Using new paradigms for teams: Platforms and technologiesSummary
8.1 Logging: Code, metrics, and results8.1.1 MLflow tracking8.1.2 Please stop printing and log your information8.1.3 Version control, branch strategies, and working with others8.2 Scalability and concurrency8.2.1 What is concurrency?8.2.2 What you can (and can’t) run asynchronouslySummary
9.1 Understanding monolithic scripts and why they are bad9.1.1 How monoliths come into being9.1.2 Walls of text9.1.3 Considerations for monolithic scripts9.2 Debugging walls of text9.3 Designing modular ML code9.4 Using test-driven development for MLSummary
10.1 ML code smells10.2 Naming, structure, and code architecture10.2.1 Naming conventions and structure10.2.2 Trying to be too clever10.2.3 Code architecture10.3 Tuple unpacking and maintainable alternatives10.3.1 Tuple unpacking example10.3.2 A solid alternative to tuple unpacking10.4 Blind to issues: Eating exceptions and other bad practices10.4.1 Try/catch with the precision of a shotgun10.4.2 Exception handling with laser precision10.4.3 Handling errors the right way10.5 Use of global mutable objects10.5.1 How mutability can burn you10.5.2 Encapsulation to prevent mutable side effects10.6 Excessively nested logicSummary
11.1 Measuring model attribution11.1.1 Measuring prediction performance11.1.2 Clarifying correlation vs. causation11.2 Leveraging A/B testing for attribution calculations11.2.1 A/B testing 10111.2.2 Evaluating continuous metrics11.2.3 Using alternative displays and tests11.2.4 Evaluating categorical metricsSummary
12.1 Detecting drift12.1.1 What influences drift?12.2 Responding to drift12.2.1 What can we do about it?12.2.2 Responding to driftSummary
13.1 Elegant complexity vs. overengineering13.1.1 Lightweight scripted style (imperative)13.1.2 An overengineered mess13.2 Unintentional obfuscation: Could you read this if you didn’t write it?13.2.1 The flavors of obfuscation13.2.2 Troublesome coding habits recap13.3 Premature generalization, premature optimization, and other bad ways to show how smart you are13.3.1 Generalization and frameworks: Avoid them until you can’t13.3.2 Optimizing too early13.4 Do you really want to be the canary? Alpha testing and the dangers of the open source coal mine13.5 Technology-driven development vs. solution-driven developmentSummary
14.1 Have you met your data?14.1.1 Make sure you have the data14.1.2 Check your data provenance14.1.3 Find a source of truth and align on it14.1.4 Don’t embed data cleansing into your production code14.2 Monitoring your features14.3 Monitoring everything else in the model life cycle14.4 Keeping things as simple as possible14.4.1 Simplicity in problem definitions14.4.2 Simplicity in implementation14.5 Wireframing ML projects14.6 Avoiding cargo cult ML behaviorSummary
15.1 Data consistency15.1.1 Training and inference skew15.1.2 A brief intro to feature stores15.1.3 Process over technology15.1.4 The dangers of a data silo15.2 Fallbacks and cold starts15.2.1 Leaning heavily on prior art15.2.2 Cold-start woes15.3 End user vs. internal use testing15.3.1 Biased testing15.3.2 Dogfooding15.3.3 SME evaluation15.4 Model interpretability15.4.1 Shapley additive explanations15.4.2 Using shapSummary
16.1 Artifact management16.1.1 MLflow’s model registry16.1.2 Interfacing with the model registry16.2 Feature stores16.2.1 What a feature store is used for16.2.2 Using a feature store16.2.3 Evaluating a feature store16.3 Prediction serving architecture16.3.1 Determining serving needs16.3.2 Bulk external delivery16.3.3 Microbatch streaming16.3.4 Real-time server-side16.3.5 Integrated models (edge deployment)Summary
A.1 What is Big O, anyway?A.1.1 A gentle introduction to complexityA.2 Complexity by exampleA.2.1 O(1): The “It doesn’t matter how big the data is” algorithmA.2.2 O(n): The linear relationship algorithmA.2.3 O(n2): A polynomial relationship to the size of the collectionA.3 Analyzing decision-tree complexityA.4 General algorithmic complexity for ML
B.1 The case for a clean experimentation environmentB.2 Containers to deal with dependency hellB.3 Creating a container-based pristine environment for experimentation

Overview

Field-tested tips, tricks, and design patterns for building machine learning projects that are deployable, maintainable, and secure from concept to production.

In Machine Learning Engineering in Action, you will learn:

Evaluating data science problems to find the most effective solution
Scoping a machine learning project for usage expectations and budget
Process techniques that minimize wasted effort and speed up production
Assessing a project using standardized prototyping work and statistical validation
Choosing the right technologies and tools for your project
Making your codebase more understandable, maintainable, and testable
Automating your troubleshooting and logging practices

Ferrying a machine learning project from your data science team to your end users is no easy task. Machine Learning Engineering in Action will help you make it simple. Inside, you’ll find fantastic advice from veteran industry expert Ben Wilson, Principal Resident Solutions Architect at Databricks.

Ben introduces his personal toolbox of techniques for building deployable and maintainable production machine learning systems. You’ll learn the importance of Agile methodologies for fast prototyping and conferring with stakeholders, while developing a new appreciation for the importance of planning. Adopting well-established software development standards will help you deliver better code management, and make it easier to test, scale, and even reuse your machine learning code. Every method is explained in a friendly, peer-to-peer style and illustrated with production-ready source code.

About the Technology
Deliver maximum performance from your models and data. This collection of reproducible techniques will help you build stable data pipelines, efficient application workflows, and maintainable models every time. Based on decades of good software engineering practice, machine learning engineering ensures your ML systems are resilient, adaptable, and perform in production.

About the Book
Machine Learning Engineering in Action teaches you core principles and practices for designing, building, and delivering successful machine learning projects. You’ll discover software engineering techniques like conducting experiments on your prototypes and implementing modular design that result in resilient architectures and consistent cross-team communication. Based on the author’s extensive experience, every method in this book has been used to solve real-world projects.

What's Inside

Scoping a machine learning project for usage expectations and budget
Choosing the right technologies for your design
Making your codebase more understandable, maintainable, and testable
Automating your troubleshooting and logging practices

About the Reader
For data scientists who know machine learning and the basics of object-oriented programming.

About the Author
Ben Wilson is Principal Resident Solutions Architect at Databricks, where he developed the Databricks Labs AutoML project. He is also an MLflow committer.

Quotes
It’s like being advised by a seasoned professional every step of the way.
- John Bassil, UNiDAYS

The ultimate resource for machine learning engineering.
- Ninoslav Cerkez, Logit

A comprehensive roadmap for implementing ML across teams and in production.
- Taylor Delehanty, Gaggle

Thorough guidance through all the steps for building a machine learning project. Full of valuable knowledge and experience.
- Rui Liu, Oracle

Great for both junior and experienced professionals.
- Ioannis Atsonios, Femtec Health