Experimentation for Engineers

8 Practical considerations

This chapter covers

• Dealing with data that does not match statistical assumptions
• Identifying biases that may creep into experiments
• Avoiding behaviors that generate false positives
• Replicating experiments to validate that their results are robust

The experimentation methods presented in this book are powerful tools that you can use to improve your engineered system. They are powerful but not foolproof. We can make subtle or simple mistakes that can cause these methods to fail.

This chapter discusses various ways in which your author—and colleagues kind enough to sit for interviews for this book—has seen these methods fail. You could read this chapter as a set of warning labels for experimental optimization.

Section ...