book

Experimentation for Engineers

by David Sweet

February 2023

Intermediate to advanced

248 pages

7h 23m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorabout the cover illustration
1.1 Examples of engineering workflows1.1.1 Machine learning engineer’s workflow1.1.2 Quantitative trader’s workflow1.1.3 Software engineer’s workflow1.2 Measuring by experiment1.2.1 Experimental methods1.2.2 Practical problems and pitfalls1.3 Why are experiments necessary?1.3.1 Domain knowledge1.3.2 Offline model quality1.3.3 SimulationSummary
2.1 Take an ad hoc measurement2.1.1 Simulate the trading system2.1.2 Compare execution costs2.2 Take a precise measurement2.2.1 Mitigate measurement variation with replication2.3 Run an A/B test2.3.1 Analyze your measurements2.3.2 Design the A/B test2.3.3 Measure and analyze2.3.4 Recap of A/B test stagesSummary
3.1 Epsilon-greedy: Account for the impact of evaluation on business metrics3.1.1 A/B testing as a baseline3.1.2 The epsilon-greedy algorithm3.1.3 Deciding when to stop3.2 Evaluating multiple system changes simultaneously3.3 Thompson sampling: A more efficient MAB algorithm3.3.1 Estimate the probability that an arm is the best3.3.2 Randomized probability matching3.3.3 The complete algorithmSummary
4.1 Optimize a single continuous parameter4.1.1 Design: Choose parameter values to measure4.1.2 Take the measurements4.1.3 Analyze I: Interpolate between measurements4.1.4 Analyze II: Optimize the business metric4.1.5 Validate the optimal parameter value4.2 Optimizing two or more continuous parameters4.2.1 Design the two-parameter experiment4.2.2 Measure, analyze, and validate the 2D experimentSummary

5.1 Model a business metric offline to make decisions online5.1.1 Model the business-metric outcome of a decision5.1.2 Add the decision-making component5.1.3 Run and evaluate the greedy recommender5.2 Explore actions with epsilon-greedy5.2.1 Missing counterfactuals degrade predictions5.2.2 Explore with epsilon-greedy to collect counterfactuals5.3 Explore parameters with Thompson sampling5.3.1 Create an ensemble of prediction models5.3.2 Randomized probability matching5.4 Validate the contextual banditSummary
6.1 Optimizing a single compiler parameter, a visual explanation6.1.1 Simulate the compiler6.1.2 Run the initial experiment6.1.3 Analyze: Model the response surface6.1.4 Design: Select the parameter value to measure next6.1.5 Design: Balance exploration with exploitation6.2 Model the response surface with Gaussian process regression6.2.1 Estimate the expected CPU time6.2.2 Estimate uncertainty with GPR6.3 Optimize over an acquisition function6.3.1 Minimize the acquisition function6.4 Optimize all seven compiler parameters6.4.1 Random search6.4.2 A complete Bayesian optimizationSummary
7.1 Focus on the business7.1.1 Don’t evaluate a model7.1.2 Evaluate the product7.2 Define business metrics7.2.1 Be specific to your business7.2.2 Update business metrics periodically7.2.3 Business metric timescales7.3 Trade off multiple business metrics7.3.1 Reduce negative side effects7.3.2 Evaluate with multiple metricsSummary
8.1 Violations of statistical assumptions8.1.1 Violation of the iid assumption8.1.2 Nonstationarity8.2 Don’t stop early8.3 Control family-wise error8.3.1 Cherry-picking increases the false-positive rate8.3.2 Control false positives with the Bonferroni correction8.4 Be aware of common biases8.4.1 Confounder bias8.4.2 Small-sample bias8.4.3 Optimism bias8.4.4 Experimenter bias8.5 Replicate to validate results8.5.1 Validate complex experiments8.5.2 Monitor changes with a reverse A/B test8.5.3 Measure quarterly changes with holdouts8.6 Wrapping upSummary
A.1 Univariate linear regressionA.2 Multivariate linear regression

Content preview from Experimentation for Engineers

Appendix C Gaussian process regression

Gaussian process regression (GPR) is a method of estimating function values directly from a set of measurements. In the context of Bayesian optimization, we imagine there exists a true function, business metric versus parameter, from which we take a few measurements. GPR estimates the value of the business metric at parameters for which we haven’t taken measurements. It forms these estimates from the measurements we have already taken.

Let’s call the parameter x (a vector of all the system parameters) and the business metric y (a scalar, a number). Let’s say we’ve already taken N measurements. We’ll index them by i and call the measurements x_i , y_i. Note that each x_i is a vector.

Our task is to estimate ...