How to Make Things Faster

Chapter 48. Delays and Throughput

Queueing and coherency delays degrade response times. Of course, they degrade throughput as well. Here’s the throughput effect of queueing delay:¹

The curve that we’d all love to be on is that straight line, where every new unit of concurrent workload you add increases throughput by as much as the previous unit you added. But the effect of queueing is to diminish the marginal utility of adding load.

Adding coherency delay degrades throughput even more. Notice how, in the following plot, the throughput curve actually bends downward as loads increase past the point labeled “peak throughput.” This downward bend creates stories like, “We added an order entry clerk, and now our system processes fewer orders per minute than before.”

This retrograde behavior happens because coherency is a manifestation of the ramp problem (which I’ll explain later). Imagine that you have ten processes that must all communicate with each other to maintain cache coherency. Then there are 9 + 8 + 7 + … + 1 = 45 communication channels that the system must maintain. If the number of processes increases just by 1, to 11, then the communication channel count increases by 10, to 55. Doubling the process count from 10 to 20 increases the communication channel count from 55 ...