O'Reilly logo

The Art of Capacity Planning by John Allspaw

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required


As we've demonstrated, with our resource ceilings pinpointed, we can predict when we'll need more of a particular resource. When we complete the task of predicting when we'll need more, we can use that timeline to gauge when to trigger the procurement process.

Your procurement pipeline is the process by which you obtain new capacity. It's usually the time it takes to justify, order, purchase, install, test, and deploy any new capacity. Figure 4-14 illustrates the procurement pipeline.

The tasks outlined in Figure 4-14 vary from one organization to another. In some large organizations, it can take a long time to gain approvals to buy hardware, but delivery can happen quickly. In a startup, approvals may come quickly, but the installation likely proceeds more slowly. Each situation will be different, but the challenge will remain the same: estimate how long the entire process will take, and add some amount of comfortable buffer to account for unforeseen problems. Once you have an idea of what that buffer timeline is, you can then work backward to plan capacity.

Typical procurement pipeline

Figure 4-14. Typical procurement pipeline

In our disk storage consumption example, we have current data on our disk consumption up to 8/15/05, and we estimate we'll run out of space on 8/30/05. You now know you have exactly two weeks to justify, order, receive, install, and deploy new storage. If you don't, you'll run out of space and be forced to trim that consumption in some way. Ideally, this two-week deadline will be long enough for you to bring new capacity online.

Procurement Time: The Killer Metric

Obviously, the when of ordering equipment is just as important as the what and how much. Procurement timelines outlined above hint at how critical it is to keep your eye on how long it will take to get what you need into production. Sometimes external influences, such as vendor delivery times and physical installation at the data center can ruin what started out to be a perfectly timed integration of new capacity.

Startups routinely order servers purely out of the fear they'll be needed. Most newly launched companies have developers to work on the product and don't need to waste money on operations-focused engineers. The developers writing the code are most likely the same people setting up network switches, managing user accounts, installing software, and wearing whatever other hats are necessary to get their company rolling. The last thing they want to worry about is running out of servers when they launch their new, awesome website. Ordering more servers as needed can be rightly justified in these cases, because the hardware costs are more than offset by the costs of preparing a more streamlined and detailed capacity plan.

But as companies mature, optimizations begin to creep in. Code becomes more refined. The product becomes more defined. Marketing starts to realize who their users are. The same holds true for the capacity management process; it becomes more polished and accurate over time.

Just-In-Time Inventory

Toyota Motors developed the first implementations of a just-in-time inventory practice. It knew there were large costs involved to organize, store, and track excess inventory of automobile parts, so it decided to reduce that "holding" inventory and determine exactly when it needed parts. Having inventory meant wasting money. Instead of maintaining a massive warehouse filled with the thousands of parts to make its cars, Toyota would only order and stock those parts as they were needed. This reduced costs tremendously and gave Toyota a competitive advantage in the 1950s. Just-in-time inventory practice is now part of any modern manufacturing effort.

The costs associated with having auto parts lying around in a warehouse can be seen analogous to having servers installed before you really need them. Rack space and power consumption in a data center cost money, as does the time spent installing and deploying code on the servers. More important, you risk suffering economically as a result of the aforementioned Moore's Law, which if your forecasts allow it, should motivate you to buy equipment later, rather than sooner.

Once you know when your current capacity will top out, and how much capacity you'll need to get through to the next cycle of procurement, you should take a few lessons from the just-in-time inventory playbook, whose sole purpose it is to eliminate waste of time and money in the process.

Here are some of the steps in our typical procurement process you'll want to pay attention to, and streamline:

  1. Determine your needs

    You know how much load your current capacity can handle, because you've followed the advice in Chapter 3 to find their ceilings and are measuring their usage constantly. Take these numbers to the curve-fitting table and start making crystal ball predictions. This is fundamental to the capacity planning process.

  2. Justify purchases

    Add some color and use attention-grabbing fonts on the graphs you just made in the previous step, because you're going to show them to the people who will approve the hardware purchases you're about to make. Spend as much time as you need to ensure your money-handling audience understands why you're asking for capacity, why you're asking for it now, and why you'll be coming back later asking for more. Be very clear in your presentations about the downsides of insufficient capacity.

  3. Solicit quotes from vendors

    Vendors want to sell you servers and storage; you want to buy servers and storage—all is balanced in the universe. Why would you choose vendor A over vendor B? Because vendor A might help alleviate some of the fear normally associated with ordering servers, through such practices as quick turnarounds on quotes and replacements, discounts on servers ordered later, or discounts tied to delivery times.

  4. Order equipment

    Can you track your order online? Do you have the phone number (gasp!) of a reliable human who can tell you where your equipment is at all times? Does the data center know the machines are coming, and have they factored that into their schedule?

  5. Physical installation

    How long will it take for the machines to make the journey from a loading dock into a rack, and cabled up to a working switch? Does the data center staff need to get involved, or are you racking machines yourself? Are there enough rack screws? Power drill batteries? Crossover cables? How long is this entire process going to take?

  6. OS/application/configuration installation

    In the next chapter, we'll talk about deployment scenarios that involve automatic OS installation, software deployment, and configuration management. However, just because it's automated doesn't mean it doesn't take time and that you shouldn't be aware of any problems that can arise.

  7. Testing

    Do you have a QA team? Do you have a QA environment? Testing your application means having some process by which you can functionally test all the bits you need to make sure everything is in its right place. Entire books are written on this topic; I'll just remind you that it's a necessary step in the journey toward production life as a server.

  8. Deploy your new equipment

    It's not over until the fat server sings. Putting a machine into production should be straightforward. When doing so, you should use the same process to measure the capacity of your new servers as outlined in the Chapter 3. Maybe you'll want to ramp up the production traffic the machine receives by increasing its weight in the load-balanced pool. If you know this new capacity relieves a bottleneck, you'll want to watch any effect that has on your traffic.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required