How to Make Things Faster

Chapter 108. Record Keeping

In 2016, Jeff and I worked on a project near Richmond, Virginia, where users were running on a system that was slower than it needed to be because they hadn’t written something down. They had done several experiments over the course of many weeks that resulted in kind of a foggy recollection of what worked and what didn’t:

• “I think I remember it being really slow when we set parameter P1=false.”

• “I think I remember it being fast when we set parameter P2=3.”

By organizing the experiments they had done into a structure called a factorial experiment matrix, we could derive more value from their work:

Parameter value		Response time
P1	P2	(seconds)
1.	true	1	284.7
2.	true	2	78.4
3.	true	3	4.2
4.	false	1
5.	false	2	9.9
6.	false	3	154.7

Since P1 had two values that they had wanted to test and P2 had three, the matrix calls for six different experiments. As you can see, they had done only five.

The structured data was harmonious with what people thought they remembered:

• “…Slow when P1=false” is confirmed in experiment 6.

• “…Fast when P2=3” is confirmed in experiment 3.

But the ultimate verdict that they had chosen, of P1=false and P2=3 resulted in the response time for their program being 154.7 seconds instead of the 4.2 seconds that was possible. And, of course, without doing another test, none of us could know whether the program would have finished even more quickly using the parameter values called for in experiment ...