How to Make Things Faster

Chapter 62. Doug

Doug was a director at a Kansas City company with exceptionally high order volumes.  One of my team’s assignments in 1994 was to draft Doug a plan for a high-availability architecture that would offer “five nines” of uptime. Getting to five nines means no more than about 5 minutes of downtime per year.

Nines	Uptime	Downtime per year
1	0.9	36d 12h
2	0.99	3d 15h 36m
3	0.999	8h 45m 36s
4	0.999 9	52m 33.6s
5	0.999 99	5m 15.36s
6	0.999 999	31.54s

I loaded up a conference room with all the people I thought we’d need. We had a great “mission specialist” present for each important aspect of the system, both functional and technical. We started the meeting around 3:00, thinking we’d be able to knock out our assignment by 5:00.

We got to four nines pretty easily. Back then, the technology was available to allow a really good team (and this was a really good team) to meet the requirement of not being down more than about 52 minutes a year. But we really got stymied when we thought about how to get from four nines to five. We knew how to reach 5 minutes of downtime a year, but we could not figure out how to do it without doubling the cost of a multimillion dollar system.

We rolled past 5:00, hoping for a eureka moment, but the eureka never came. We kept trying, but we just couldn’t come up with an economical solution. Finally, around 7:00, we decided our only options were either to die in that ...