How to Make Things Faster

Chapter 80. The Buried Outlier

Something like this is eventually going to happen to you:

You: “Your order entry program was slow on Wednesday because it executed a single 512-byte write call that took 18.6 seconds.”

Businessperson: “Oh wow, I’ll share that information with our storage administrator!”

Storage administrator: “No, that’s impossible. We couldn’t have had an 18.6-second single-block write call. Our average write latency on Wednesday was 0.001 seconds.”

You: “Well, the pwrite call is in the program’s trace. It completed Wednesday at 14:02:28.176910 and took 18.582771 seconds.”

SA: “That couldn’t have happened. Your trace is wrong.”

Yet your trace is correct. How is that possible?

There are lots of ways it could have happened. For example, if there were a million pwrite calls on Wednesday, and every call took 0.001 seconds apiece except for the one that took 18.6 seconds, then the average would be 0.001 019 seconds per call.¹ In a million calls, you can bury five 18.6-second outliers without increasing the average by even a tenth of a millisecond.²

It’s easy for outliers to get buried, especially in really long lists, so just because a number seems implausible doesn’t mean it didn’t happen.

¹ ((1 call) × 18.6 (seconds/call) + (1,000,000 − 1) calls × 0.001(seconds/call)) / 1,000,000 calls ≈ 0.001 019 seconds/call.

² ((5 calls) × 18.6 (seconds/call) + (1,000,000 − 5) calls × 0.001(seconds/call)) / 1,000,000 calls ≈ 0.001 093 seconds/call.