book

Experimentation for Engineers

Name: Experimentation for Engineers
Author: David Sweet
ISBN: 9781617298158

by David Sweet

February 2023

Intermediate to advanced

248 pages

7h 23m

English

Manning Publications

Audiobook available

Read now

Unlock full access

inside front cover
Experimentation for Engineers
Copyright
dedication
contents
front matter
prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1 Optimizing systems by experiment
1.1 Examples of engineering workflows1.1.1 Machine learning engineer’s workflow1.1.2 Quantitative trader’s workflow1.1.3 Software engineer’s workflow1.2 Measuring by experiment1.2.1 Experimental methods1.2.2 Practical problems and pitfalls1.3 Why are experiments necessary?1.3.1 Domain knowledge1.3.2 Offline model quality1.3.3 SimulationSummary
2 A/B testing: Evaluating a modification to your system
2.1 Take an ad hoc measurement2.1.1 Simulate the trading system2.1.2 Compare execution costs2.2 Take a precise measurement2.2.1 Mitigate measurement variation with replication2.3 Run an A/B test2.3.1 Analyze your measurements2.3.2 Design the A/B test2.3.3 Measure and analyze2.3.4 Recap of A/B test stagesSummary
3 Multi-armed bandits: Maximizing business metrics while experimenting
3.1 Epsilon-greedy: Account for the impact of evaluation on business metrics3.1.1 A/B testing as a baseline3.1.2 The epsilon-greedy algorithm3.1.3 Deciding when to stop3.2 Evaluating multiple system changes simultaneously3.3 Thompson sampling: A more efficient MAB algorithm3.3.1 Estimate the probability that an arm is the best3.3.2 Randomized probability matching3.3.3 The complete algorithmSummary
4 Response surface methodology: Optimizing continuous parameters
4.1 Optimize a single continuous parameter4.1.1 Design: Choose parameter values to measure4.1.2 Take the measurements4.1.3 Analyze I: Interpolate between measurements4.1.4 Analyze II: Optimize the business metric4.1.5 Validate the optimal parameter value4.2 Optimizing two or more continuous parameters4.2.1 Design the two-parameter experiment4.2.2 Measure, analyze, and validate the 2D experimentSummary

5 Contextual bandits: Making targeted decisions
5.1 Model a business metric offline to make decisions online5.1.1 Model the business-metric outcome of a decision5.1.2 Add the decision-making component5.1.3 Run and evaluate the greedy recommender5.2 Explore actions with epsilon-greedy5.2.1 Missing counterfactuals degrade predictions5.2.2 Explore with epsilon-greedy to collect counterfactuals5.3 Explore parameters with Thompson sampling5.3.1 Create an ensemble of prediction models5.3.2 Randomized probability matching5.4 Validate the contextual banditSummary
6 Bayesian optimization: Automating experimental optimization
6.1 Optimizing a single compiler parameter, a visual explanation6.1.1 Simulate the compiler6.1.2 Run the initial experiment6.1.3 Analyze: Model the response surface6.1.4 Design: Select the parameter value to measure next6.1.5 Design: Balance exploration with exploitation6.2 Model the response surface with Gaussian process regression6.2.1 Estimate the expected CPU time6.2.2 Estimate uncertainty with GPR6.3 Optimize over an acquisition function6.3.1 Minimize the acquisition function6.4 Optimize all seven compiler parameters6.4.1 Random search6.4.2 A complete Bayesian optimizationSummary
7 Managing business metrics
7.1 Focus on the business7.1.1 Don’t evaluate a model7.1.2 Evaluate the product7.2 Define business metrics7.2.1 Be specific to your business7.2.2 Update business metrics periodically7.2.3 Business metric timescales7.3 Trade off multiple business metrics7.3.1 Reduce negative side effects7.3.2 Evaluate with multiple metricsSummary
8 Practical considerations
8.1 Violations of statistical assumptions8.1.1 Violation of the iid assumption8.1.2 Nonstationarity8.2 Don’t stop early8.3 Control family-wise error8.3.1 Cherry-picking increases the false-positive rate8.3.2 Control false positives with the Bonferroni correction8.4 Be aware of common biases8.4.1 Confounder bias8.4.2 Small-sample bias8.4.3 Optimism bias8.4.4 Experimenter bias8.5 Replicate to validate results8.5.1 Validate complex experiments8.5.2 Monitor changes with a reverse A/B test8.5.3 Measure quarterly changes with holdouts8.6 Wrapping upSummary
Appendix A Linear regression and the normal equations
A.1 Univariate linear regressionA.2 Multivariate linear regression
Appendix B One factor at a time
Appendix C Gaussian process regression
index
inside back cover

Content preview from Experimentation for Engineers

3 Multi-armed bandits: Maximizing business metrics while experimenting

This chapter covers

Defining the multi-armed bandit (MAB) problem
Modifying A/B testing’s randomization procedure
Extending epsilon-greedy to simultaneously evaluate multiple system changes
Evaluating system changes even more quickly with Thompson sampling

In the previous chapter, we learned how to use A/B testing to evaluate changes to the system your engineering team is building. Once the tooling is in place to run A/B tests, the team should see a steady increase in the quality of the system as new changes follow the engineering workflow: implement a change candidate, evaluate it offline, and evaluate it online with an A/B test.

As the use of A/B testing increases, you’ll ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781617298158Publisher Support Publisher Website

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Experimentation for Engineers

by David Sweet

3 Multi-armed bandits: Maximizing business metrics while experimenting

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.