Chapter 1. Understanding, Measuring, and Improving Your Availability
No one cares whether your system has great features if they can’t use it.
One of the most important topics in architecting for scalable systems is availability. Although there are some companies and some services for which a certain amount of downtime is reasonable and expected, most businesses cannot have any downtime at all without it affecting their customers’ satisfaction, and ultimately the company’s bottom line.
The following are fundamental questions that all companies must ask as they determine how important system availability is to themselves and their customers. It is these questions, and the inevitable answers to them, that are the core of why availability is critical to highly scaled applications.
- Why buy from you?
Why should someone buy your service if it is not operational when they need it?
- What do your customers think?
What do your customers think or feel when they need to use your service and it’s not operational?
- How do you make customers happy?
How can you make your customers happy, make your company money, and meet your business promises and requirements if your service is down?
Keeping your customers happy and engaged with your system is possible only if your system is operational. There is a direct and meaningful correlation between system availability and customer satisfaction.
High availability is such a critical component of building highly scalable systems that we will devote a significant amount of time to the topic in this book. How do you build a system (a service or application or environment) that is highly available even when a wide range of demands are placed on it?
Availability Versus Reliability
Reliability, in our context, generally refers to the quality of a system. Typically, it means the ability of a system to consistently perform according to specifications. You speak of software as reliable if it passes its test suites and does generally what you think it should do. Reliability answers the question:
“Is the response to my query correct?”
Availability, in our context, generally refers to the ability of your system to perform the tasks it is capable of doing. Is the system around? Is it operational? Is it responding? If the answer is “yes,” it is available. Availability answers the questions:
“Am I getting a response?”
“Did the response arrive in time?”
As you can see, availability and reliability are very similar. It is hard for a system to be available if it is not also reliable, and it is hard for a system to be reliable if it is not also available.
More formally, here is what we mean when we use these terms:
The ability of your system to perform the operations it is intended to perform without making a mistake.
The ability of your system to be operational when needed in order to perform those operations.
A system that adds 2 + 3 and gets 6 has poor reliability. A system that adds 2 + 3 and never returns a result at all has poor availability. Reliability can often be fixed by testing. Availability is usually much harder to solve.
You can introduce a software bug in your application that can cause 2 + 3 to produce the answer 6. This can be easily caught and fixed in a test suite.
However, assume you have an application that reliably produces the result 2 + 3 = 5. Now imagine running this application on a computer that has a flaky network connection. The result? You run the application, and sometimes it returns 5, and sometimes it doesn’t return anything. The application may be reliable, but it is not available.
We will focus almost exclusively on architecting highly available systems. We will assume your system is reliable, we will assume you know how to build and run test suites, and we will discuss reliability only when it has a direct impact on your system architecture or its availability.
What Causes Poor Availability?
- Resource exhaustion
- Unplanned load-based changes
Increases in the popularity of your application might require code and application changes to handle the increased load. These changes, often implemented quickly and at the last minute with little or no forethought or planning, increase the likelihood of problems occurring.
- Increased number of moving parts
As an application gains popularity, it is often necessary to assign more and more developers, designers, testers, and other individuals to work on and maintain it. This larger number of individuals working on the application creates a large number of moving parts, whether those moving parts are new features, changed features, or just general application maintenance. The more individuals working on the application, the more moving parts within the application and the greater the chance for bad interactions to occur in it.
- Outside dependencies
The more dependencies your application has on external resources, such as SaaS services, infrastructure, or cloud-based services, the more it is exposed to availability problems caused by those resources.
- Technical debt
Increases in the applications complexity typically increase technical debt (i.e., the accumulation of desired software changes and pending bug fixes that often build up over time as an application grows and matures). Technical debt increases the likelihood of a problem occurring.
All fast-growing applications have one, some, or all of these problems. As such, potential availability problems can begin occurring in applications that previously performed flawlessly. The problems can quietly creep up on you, or the problems may start suddenly without warning.
But most growing applications will eventually begin having availability problems.
Availability problems cost you money, they cost your customers money, and they cost you your customers’ trust and loyalty. Your company cannot survive for long if you constantly have availability problems.
Building applications designed to scale means building applications designed for high availability.
Measuring availability is important to keeping your system highly available. Only by measuring availability can you understand how your application is performing now and examine how your application’s availability changes over time.
The most widely held mechanism for measuring the availability of a web application is calculating the percent of time it’s accessible for use by customers. We can describe this by using the following formula for a given period:
Site availability percentage = total_seconds_in_period − seconds_system_is_down / total_seconds_in_period
Let’s consider an example. Suppose that over the month of April, your website was down twice; the first time it was down for 37 minutes, and the second time it was down for 15 minutes. What is the availability of your website?
You can see from the following example that it takes only a small amount of outage to have an impact on your availability percentage:
Total number of seconds down = (37 + 15) × 60 = 3,120 s
Total number of seconds in month = 30 days × 86,400 s/day = 2,592,000 s
Site availability percentage = total_seconds_in_period − seconds_system_is_down / total_seconds_in_period
Site availability percentage = 2,592,000 s − 3,120 s / 2,592,000 s
Site availability percentage = 99.8795
Your site availability is 99.8795%.
Often you will hear availability described as “the nines.” This is a shorthand way of indicating high-availability percentages. Table 1-1 illustrates what it means. An application that has “2 nines” availability must be available 99% of the time. This means in a typical month it can be down for 432 minutes and still meet the 99% available goal. By contrast, a “4 nines” application must be available 99.99% of the time, meaning it can be down a mere four minutes in a typical month.
|2 nines||99%||432 minutes|
|3 nines||99.9%||43 minutes|
|4 nines||99.99%||4 minutes|
|5 nines||99.999%||26 seconds|
|6 nines||99.9999%||2.6 seconds|
In the preceding example, we see that the website has fallen just short of the 3 nines metric (99.8795% compared to 99.9%). For a website that maintains 5 nines of availability, there can be only 26 seconds of downtime every month.
What’s a reasonable availability number in order to consider your system as high availability? It is impossible to give a single answer to this question because it depends dramatically on your website, your customer expectations, your business needs, and your business expectations. You need to determine for yourself what number is required for your business.
Often, for basic web applications, 3 nines is considered acceptable availability. Using Table 1-1, this amounts to 43 minutes of downtime every month.
Planned Outages Are Still Outages
Here’s a comment that I often overhear: “Our application never fails. That’s because we regularly perform system maintenance. By scheduling weekly two-hour maintenance windows and performing maintenance during these windows, we keep our availability high.”
Does this group keep its application’s availability high?
Let’s find out:
Site availability percentage = total_hours_in_period − hours_system_is_down / total_hours_in_period
hours_in_week = 7 days × 24 hours = 168 hours
hours_unavailable_each_week = 2 hours
Site availability (no failures) = 168 hours − 2 hours / 168 hours = 98.8%
Site availability (no failures) = 98.8%
Without having a single failure of its application, the best this organization can achieve is 98.8% availability. This falls short of even 2 nines availability (98.8% versus 99%).
Planned maintenance hurts nearly as much as unplanned outages. If your customer needs your application to be available and it isn’t, your customer has a negative experience. It doesn’t matter whether or not you planned for the outage.
Improving Your Availability When It Slips
Your application is operational and online. Your systems are in place, and your team is operating efficiently. Everything seems to be going well. Your traffic is steadily increasing, and your sales organization is very happy. All is well.
Then there’s a bit of a slip. Your system suffers an unanticipated outage. But that’s OK; your availability has been fantastic until now. A little outage is no big deal. Your traffic is still increasing. Everyone shrugs it off—it was just “one of those things.”
Then it happens again—another outage. Oops. Well, OK. Overall, we’re still doing well. No need to panic; it was just another “one of those things.”
Then another outage...
Now your CEO is a bit concerned. Customers are beginning to ask what’s going on. Your sales team is starting to worry.
Then another outage...
Suddenly, your once stable and operational system is becoming less and less stable; your outages are getting more and more attention.
Now you’ve got real problems.
What happened? Keeping your system highly available is a daunting task. What do you do if availability begins to slip? What do you do if your application availability has fallen or begins to fall, and you need to improve it to keep your customers satisfied?
Knowing what you can do when your availability begins to slip will help you to avoid falling into a vicious cycle of problems. What can you do to avoid your availability slipping? Some key things are:
Measure and track your current availability
Automate your manual processes
Automate your deployment processes
Maintain and track all configurations in a management system
Allow quick changes and experiments, with an easy rollback capability if a problem occurs
Aim to continuously improve your applications and systems
Keep on top of availability as a core issue as your application changes and grows
The following sections detail these key steps in further detail.
Measure and Track Your Current Availability
To understand what is happening to your availability, you must first measure what your current availability is. Tracking when your application is or is not available gives you an availability percentage that can show how you are performing over a specific period of time. You can use this to determine whether your availability is improving or faltering.
You should continuously monitor your availability percentage and report the results on a regular basis. On top of this, overlay key events in your application, such as when you performed system changes and improvements. This way you can see whether there is a correlation over time between system events and availability issues. This can help you to identify risks to your availability.
Next, you must understand how your application can be expected to perform from an availability standpoint. A tool that you can use to help manage your application availability is service tiers. These are simply labels associated with services that indicate how critical a service is to the operation of your business. The use of service tiers allows you and your teams to distinguish between mission-critical services and those that are valuable but not essential. We’ll discuss service tiers in more depth in Chapter 7.
Finally, create and maintain a risk matrix. With this tool, you can gain visibility into the technical debt and associated risk present in your application. Risk matrices are covered more fully in Chapter 9.
Now that you have a way to track your availability and a way of identifying and managing your risk, you will want to review your risk management plans on a regular basis.
Additionally, you should create and implement mitigation plans to reduce your application risks. This will give you a concrete set of tasks you and your development teams can implement to tackle the riskiest parts of your application. This is discussed in detail in Chapter 9.
Automate Your Manual Processes
You should never perform a manual operation on a production system.
When you make a change to your system, the change might improve your system, or it might compromise it. Using only repeatable tasks gives you the following:
The ability to test a task before implementing it. Testing what happens when you make a specific change is critical to avoiding mistakes that cause outages.
The ability to tweak the task to perform exactly what you want it to do. This lets you implement improvements to the change you are about to make before you actually make the change.
The ability to have the task reviewed by a third party. This increases the likelihood that the task will have no unexpected side effects.
The ability to put the task under version control. Version control systems allow you to determine when the task is changed, by whom, and for what reasons.
The ability to apply the task to related resources. Making a change to a single server that improves how that server works is great. Being able to apply the same change to every affected server in a consistent way makes the task even more useful.
The ability to have all related resources act consistently. If you continuously make “one-off” changes to resources such as servers, the servers will begin to drift and act differently from one another. This makes it difficult to diagnose problematic servers because there will be no baseline of operational expectation that you can use for comparison.
The ability to implement repeatable tasks. Repeatable tasks are auditable tasks. Auditable tasks are tasks that you can analyze later for their impact, positive or negative, on the system as a whole.
There are many systems for which no one has access to the production environment. Period. The only access to production is through automated processes and procedures. The owners of these systems lock down their environments like this specifically for the aforementioned reasons.
In summary, if you can’t repeat a task, it isn’t a useful task. There are many places where adding repeatability to changes will help keep your system and application stable. This includes implementing server configuration changes, making performance-tuning tweaks and adjustments, restarting servers, restarting jobs and tasks, changing routing rules, and upgrading and deploying software packages. We’ll now look at some examples of repeatable tasks you should employ.
At the very least, write a script that will make the change, and then check that script into your software change management system. This enables you to make the same change to all servers in your system uniformly. Additionally, when you need to add a new server to your system or replace an old one, having a known configuration that can be applied improves the likelihood that you can add the new server to your system safely, with minimal impact.
But even better—and consistent with modern, state of the art, configuration management best practices—is to employ a concept called Infrastructure as Code. Infrastructure as Code involves describing your infrastructure in a standard, machine-readable specification and then passing that specification through an infrastructure tool that will create and/or update your infrastructure and your configuration to match the specification. Tools like Puppet and Chef can help make this process easier to manage.
Then you take this specification and check it into your version control system, so that changes to the specification can be tracked just like code changes can be tracked. Running the specification through the infrastructure tool anytime a change is made to the specification will update your live infrastructure to match the specification.
If anyone needs to make a change to the infrastructure or its configuration, they must make the change to the specification, check the change into version control, and then “deploy” the change via the infrastructure tool to update your live infrastructure to match. In this manner, you can:
Ensure all components of the infrastructure have a consistent, known, and stable configuration.
Track all changes to the infrastructure so they can be rolled back if needed, or used to assist in correlation with system events and outages.
Allow a peer review process, similar to a code review process, to ensure changes to your infrastructure are correct and appropriate.
Allow creating duplicate environments to assist in testing, staging, and development with an environment identical to production.
This same sort of process applies to all infrastructure components. This includes not only servers and their operating system configuration but also other cloud components, VPCs, load balancers, switches, routers, network components, and monitoring applications and systems.
For Infrastructure as Code management to be useful, it must be employed for all system changes, all the time. It is never acceptable to bypass the infrastructure management system to make a change, no matter the circumstances. Not ever.
You would be surprised the number of times I have received an operational update email that said something like, “We had a problem with one of our servers last night. We hit a limit to the maximum number of open files the server could handle. So I tweaked the kernel variable and increased the maximum number of open files, and the server is operational again.”
That is, it is operating correctly until someone accidentally overwrites the change because there was no documentation of the change. Or until one of the other servers running the application has the same problem but did not have this change applied to it.
Or someone makes another change, which breaks the application because it is inconsistent with the undocumented change you just made.
Consistency, repeatability, and unfaltering attention to detail are critical to making a configuration management process work. And a standard and repeatable configuration management process such as we describe here is critical to keeping your scaled system highly available.
Change experiments and high frequency changes
Another advantage of having a highly repeatable, highly automated process for making changes and upgrades to your system is that it allows you to experiment with changes. Suppose that you have a configuration change you want to make to your servers that you believe will improve their performance in your application. By using an automated change management process, you can do the following:
Document your proposed change.
Review the change with people who are knowledgeable and might be able to provide suggestions and improvements.
Test the change on servers in a test or staging environment.
Deploy your change quickly and easily.
Examine the results quickly. If the change didn’t have the desired results, you can quickly roll back to a known good state.
The keys to implementing this process are to have an automated change process with rollback capabilities, and to have the ability to make small changes to your system easily and often.1 The former lets you make changes consistently; the latter lets you experiment and roll back failed experiments with little to no impact on your customers.
Automated change sanity testing
By having an automated change and deploy process,2 you can implement an automated sanity test of all changes. You can use a browser testing application for web applications or use a synthetic monitoring system to simulate customer interaction with your application.
When you are ready to deploy a change to production, you can have your deployment system first automatically deploy the change to a test or staging environment. You can then have these automated tests run and validate that the changes did not break your application.
If and when those tests pass, you can automatically deploy the change in a consistent manner to your production environment. Depending on how your tests are constructed, you should be able to run the tests regularly on your production environment as well to validate that no changes break anything there.
Improve Your Systems
Now that you have a system to monitor availability, a way to track risk and mitigations in your system, and a way to easily and safely apply consistent changes to your system, you can focus your efforts on improving the availability of your application itself.
Regularly review your risk matrix and your recovery plans. Make reviewing them part of your postmortem process. Execute projects that are designed to mitigate the risks identified in your matrix. Roll out those changes in an automated and safe way, using the sanity tests discussed earlier. Examine how the mitigation has improved your availability. Continue the process until your availability reaches the level you want and need it to be.
Keep on Top of Availability in Your Changing and Growing Application
As your system grows, you’ll need to handle larger and larger traffic and data demands. Much of the content in this book is designed to help you address application availability and scalability issues as your application grows and changes. In particular, managing mistakes and errors at scale is discussed in Chapter 2. Service tiers, which you can use to identify key availability-impacting services, are discussed in Chapter 7. And service-level agreement (SLA) management is discussed in Chapter 8.
Typically, your application will change continuously. As such, your risks, mitigations, contingencies, and recovery plans need to constantly change.
Five Focuses to Improve Application Availability
Building a scalable application that has high availability is not easy and does not come automatically. Problems can crop up in unexpected ways that can cause your beautifully functioning application to stop working for all or some of your customers.
These availability problems often arise from the areas you least expect, and some of the most serious availability problems can originate from extremely benign sources.
No one can anticipate where problems will come from, and no amount of testing will find all issues. Many of these are systemic problems, not merely code problems.
To find these availability problems, we need to step back and take a systemic look at your application and how it works. Here are five things you can and should focus on when building a system to make sure that, as its use scales upwards, availability remains high:
Build with failure in mind
Always think about scaling
Respond to availability issues in a predictable and defined way
Let’s look at each of these individually.
Focus #1: Build with Failure in Mind
As Werner Vogels, CTO of Amazon, says, “Everything fails all the time.” Plan on your applications and services failing. It will happen. Now, deal with it.
Assuming your application will fail, how will it fail? As you build your system, consider availability concerns during all aspects of your system design and construction.
What design constructs and patterns have you considered or are you using that will help improve the availability of your software?
Using design constructs and patterns, such as simple error catching deep within your application, retry logic, and circuit breakers, allows you to catch errors when they have affected the smallest available subset of functionality. This allows you to limit the scope of a problem and have your application still provide useful capabilities even if part of the application is failing.
What do you do when a component you depend on fails? How do you retry? What do you do if the problem is an unrecoverable (hard) failure, rather than a recoverable (soft) failure?
Circuit breaker patterns are particularly useful for handling dependency failures because they can reduce the impact a dependency failure has on your system. Without a circuit breaker, you can decrease the performance of your application because of a dependency failure (e.g., because an unacceptably long timeout is required to detect the failure). With a circuit breaker, you can “give up” and stop using a dependency until you are certain that dependency has recovered.
What do you do when a component that is a customer of your system behaves poorly? Can you handle excessive load on your system? Can you throttle excessive traffic? Can you handle garbage data passed in? What about excessive data?
Sometimes denial-of-service attacks can come from “friendly” sources. For example, a customer of your application may see a sudden surge in activity that requires a significant increase in the volume of requests to your application. Alternatively, a bug in your customer’s application may cause them to call your application at an unacceptably high rate. What do you do when this happens? Does the sudden increase in traffic bring your application down? Or can you detect this problem and throttle the request rate, limiting or removing the impact to your application?
Focus #2: Always Think About Scaling
Just because your application works now does not mean it will work tomorrow. Most web applications have increasing traffic patterns. A website that generates a certain amount of traffic today might generate significantly more traffic sooner than you anticipate. As you build your system, don’t build it for today’s traffic; build it for tomorrow’s traffic.
Specifically, this might mean:
Architect in the ability to increase the size and capacity of your databases.
Think about what logical limits exist to your data scaling. What happens when your database tops out in its capabilities? Identify and remove these limits before your usage approaches them.
Build your application so that you can add additional application servers easily. This often involves being observant of where and how state is maintained and of how traffic is routed.
Redirect static traffic to offline providers. This allows your system to deal only with the dynamic traffic that it is designed to deal with. Using external content delivery networks (CDNs) not only can reduce the traffic your network has to handle but also allows the efficiencies of scale that CDNs provide to get that static content to your customers more quickly.
Think about whether specific pieces of dynamic content can actually be generated statically. Often, content that appears dynamic is actually mostly static, and the scalability of your application can be increased by making this content static. This “dynamic that can be static” data is sometimes hidden where you don’t expect it.
Focus #3: Mitigate Risk
Keeping a system highly available requires removing risk from the system. Often the cause of a system failure could have been identified as a risk before the failure actually occurred. Identifying risk is a key method of increasing availability. All systems have risk in them. There is risk that:
A server will crash
A database will become corrupted
A returned answer will be incorrect
A network connection will fail
A newly deployed piece of software will fail
Keeping a system available requires removing risk. But as systems become more and more complicated, this becomes less and less possible. Keeping a large system available is more about managing what your risk is, how much risk is acceptable, and what you can do to mitigate that risk.
We call this risk management. We will be talking extensively about risk management in Chapter 9. Risk management is at the heart of building highly available systems.
Part of risk management is risk mitigation. Risk mitigation is knowing what to do when a problem occurs in order to reduce the impact of the problem as much as possible. Mitigation is about making sure your application works as well and as completely as possible, even when services and resources fail. Risk mitigation requires thinking about the things that can go wrong and putting a plan together now so that you will be able to handle the situation when it does happen.
The preceding sidebar is an example of risk mitigation; the process of identifying the risk, determining what to do, and implementing these mitigations is risk management.
This process will often uncover unknown problems in your application that you will want to fix immediately instead of waiting for them to occur. It also can create processes and procedures to handle known failure modes so that the cost of that failure is reduced in duration or severity.
Availability and risk management go hand in hand. Building a highly available system is significantly about managing risk.
Focus #4: Monitor Availability
You can’t know if there is a problem in your application unless you can see a problem. Make sure your application is properly instrumented so that you can see how the application is performing from an external perspective as well as by way of internal monitoring.
Proper monitoring depends on the specifics of your application and needs, but it usually entails some of the following capabilities:
- Server monitoring
To monitor the health of your servers and make sure they keep operating efficiently.
- Configuration change monitoring
To monitor your system configuration and identify if and when changes to your infrastructure impact your application.
- Application performance monitoring
To look inside your application and services to make sure they are operating as expected.
- Synthetic testing
To examine in real time how your application is functioning from the perspective of your users, in order to catch problems customers might see before they actually see them.
To inform appropriate personnel when a problem occurs so that it can be quickly and efficiently resolved, minimizing the impact on your customers.
There are many good monitoring systems available, both free and paid services. I personally recommend New Relic. It provides all of the aforementioned monitoring and alerting capabilities. As a Software as a Service (SaaS) offering, it can support your monitoring needs at pretty much any scale your application may require.
After you have started monitoring your application and services, start looking for trends in your performance. When you have identified the trends, you can look for outliers and treat them as potential availability issues. You can use these outliers by having your monitoring tools send you an alert when they are identified, before your application fails. Additionally, you can track as your system grows and make sure your scalability plan will continue to work.
Establish internal, private operational goals for service-to-service communications, and monitor them continuously. This way, when you see a performance- or availability-related problem, you can quickly diagnose which service or system is responsible and address the problem. Additionally, you can see “hot spots”—areas in which your performance is not what it could be—and put development plans in place to address these issues.
Focus #5: Respond to Availability Issues in a Predictable and Defined Way
Monitoring systems are useless unless you are prepared to act on the issues that arise. This means being alerted when problems occur so that you can take action. Additionally, you should establish processes and procedures that your team can follow to help diagnose issues and easily fix common failure scenarios.
For example, if a service becomes unresponsive, you might have a set of remedies to try to make the service responsive. This might include tasks such as running a test to help diagnose where the problem is, restarting a daemon that is known to cause the service to become unresponsive, or rebooting a server if all else fails. Having standard processes in place for handling common failure scenarios will decrease the amount of time your system is unavailable. Additionally, these processes can provide useful follow-up diagnostic information to your engineering teams to help them deduce the root cause of common ailments.
When an alert is triggered for a service, the owners of that service must be the first ones alerted. They are, after all, the ones responsible for fixing any issues with their service. However, other teams who are closely connected to the troubled service and depend on it might also want to be alerted of problems when they occur. For example, if a team makes use of a particular service, they may want to know when that service fails so that they can potentially be more proactive in keeping their systems active during the dependent service outage.
These standard processes and procedures should be part of an online support manual available to all team members who handle on-call responsibility. These support artifacts should also contain contact lists for owners of related services and systems as well as contacts to call to escalate the problem if a simple solution is not possible. There are SaaS applications available that can automate the management and versioning of these support documents and make them available on demand during events.
All of these processes, procedures, and support manuals should be prepared ahead of time so that during an outage your on-call personnel know exactly what to do in various circumstances to restore operations quickly. These processes and procedures are especially useful because outages often occur during inconvenient times, such as the middle of the night or weekends—times when your on-call team might not perform at peak mental efficiency. These recommendations will assist your team in making smarter and safer moves toward restoring your system to operational status.
No one can anticipate where and when availability issues will occur. But you can assume that they will occur, especially as your system scales to larger customer demands and more complex applications. Being prepared in advance to handle availability concerns is the best way to reduce the likelihood and severity of problems. The information in this chapter, including the five focuses, offers a solid strategy for keeping your applications highly available.