So far I’ve avoided talking too much about measurements, despite their obvious importance to optimization. Why? Because in most cases people get it right. Your intuition nudges you to run the code several times, measure results, and pick the most commonly seen measurement.
This is a viable approach, and it should yield meaningful results most of the time. But if you get your measurements wrong, your optimization will go wrong too. You may either miss the small speedup, or falsely believe in ten times optimization where in fact there is none.
So let’s think about what can go wrong with measurements and how to deal with it.
External factors are the first issue that comes to mind. These might be processes running in parallel ...