O'Reilly logo

Effective Performance Engineering by Todd DeCapua, Shane Evans

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 1. Getting Started

How do we get started with Effective Performance Engineering?

Let’s start by defining it. When speaking with different individuals and organizations, we’ve found the definition of “Effective Performance Engineering” or “Performance Engineering” varies greatly, so we wanted to define it upfront.

Performance Engineering represents a cultural shift in the way organizations view their essential processes. It embraces practices and capabilities that build quality and performance throughout an organization. This enables organizations to increase revenue, customer attraction and retention, brand value, and competitive advantage—all while focusing on meeting and exceeding the expectations of their end users.

Let’s go back to see where performance was first introduced in the modern computer era in technology history: when the Z1 was created by German Konrad Zuse in his parents’ living room between 1936 and 1938. The Z1 is considered to be the first electro-mechanical binary programmable computer, and the first really functional modern computer. This marked the start of “Performance Engineering” for hardware and software related to the modern computer, some nearly 80 years ago.

Prior to the first functional modern computer, there are many examples of other performance-related topics associated with crops, livestock, medicine, mechanics, and plenty more. As with anything the challenge remains similar, but the practices and capabilities change. Many of these practices were handed down from mentor to mentee and through apprenticeship or learned through individual real-world experiences.

We will take a look at the Effective Performance Engineering of today, and how it relates to future technology and business capabilities (including computer and associated hardware and software) to serve the end user, and several other impacting factors and opportunities as we continue to accelerate.

What Is Effective Performance Engineering?

While Performance Engineering is often defined narrowly as ensuring that nonfunctional requirements are met (such as response times, resource utilization, and throughput), the trend has moved toward a much broader application of the term.

“Performance Engineering” doesn’t refer only to a specific role. More generally, it refers to the set of skills and practices that are gradually being understood and adopted across organizations that focus on achieving higher levels of performance in technology, in the business, and for end users.

Performance Engineering embraces practices and capabilities that build in quality and performance throughout an organization, including functional requirements, security, usability, technology platform management, devices, third-party services, the cloud, and more.

Stakeholders of Performance Engineering run the gamut, including Business, Operations, Development, Testing/Quality Assurance, and End Users.

We’ll explore several different facets of Performance Engineering in this book, providing a well-rounded overview of the practices and capabilities that make up Effective Performance Engineering.


The traditional goal of Performance Engineering between the 1970s and the late 90s was to optimize the application hardware to suit the needs of the business or, more accurately, the IT organization that provided the services to the business. This activity was really more of a capacity planning function, and many teams charged with carrying the mantle of performance reported to operations or infrastructure teams. Some still do (and that’s okay).

As hardware became more commoditized and the adoption of virtual infrastructure and “the cloud” more prevalent, this function took a backseat to development in an effort to deliver business applications and changes faster. It isn’t uncommon now for teams to have multiple environments to support development, test, production, and failover. While certainly more cost-effective than ever, virtualization has given us the false sense that these environments are free.

The cloud allows service providers to charge a premium for computer power in exchange for the promise of higher uptime, higher availability, and virtually unlimited capacity. However, the cloud doesn’t promise an optimal user experience. Applications need to be optimized for the cloud in order to maximize the potential return on investment.


Over the last 30 years, software has transformed from monolithic to highly distributed, and even the concept of model-view-controller (MVC) has evolved to service-oriented (SOA) and micro-service architectures, all in an effort to reduce the number of points of change or failure, and improve the time-to-value when new functionality is implemented. Isolating components also allows developers to test their discrete behavior, which often leaves end-to-end integrated testing out of scope. The assumption here is that if every component behaves as it should, the entire system should perform well. In an isolated environment this may be true, but there are many factors introduced when you’re building large-scale distributed systems that impact performance and the end-user experience—factors that may not be directly attributed to software, but should be considered nonetheless, such as network latency, individual component failures, and client-side behavior. It is important to build and test application components with all of these factors represented in order to optimize around them.


Every organization and group has a mission and vision. While they strive to attain these goals, performance becomes implied or implicit. But performance needs to be a part of all decisions around the steps taken to achieve a goal; it forms the basis of how an organization will embody Performance Engineering throughout their culture to achieve their mission and vision.

We need to treat performance as a design principle, similar to deciding whether to build applications using MVC or micro-services architectures, or asking why a new epic (or the relative size of a requirement, in Agile terminology) is important to the business, and how performance with the business/technology/end user will make a difference for all stakeholders. Performance needs to be an overarching requirement from the beginning, or we have already started on the wrong foot.

In order to build a culture that respects the performance requirements of the organization and our end users, there needs to be some incentive to do so. If it doesn’t come from the top down, then we can take a grassroots approach, but first we need to quantify what performance means to our business, users, and team. We must understand the impact and cost of every transaction in the system, and seek to optimize that for improved business success.

Throughout this book are sidebars in which we look at five companies and examine how performance is built in to their culture. These organizations are from a variety of industries and verticals, showing the diversity of how a performance culture drives everything a business does, enabling it to deliver amazing results. Here are the five companies we will look at in more detail:

  • Google

  • Wegmans

  • DreamWorks

  • Salesforce

  • Apple

The key takeaway here should be that performance is everyone’s responsibility, not just the developers', the testers', or the operations team’s. It needs to be part of our collective DNA. “Performance First” can be a mantra for every stakeholder.

The end-user experience should be at the forefront of thinking when it comes to performance. The satisfaction of your end users will ultimately drive business success (or failure), and can be quantified by a number of metrics in described in “Metrics for Success”. The point is that it shouldn’t matter whether your servers can handle 1,000 hits/second or if CPU usage is below 80%. If the experience of the end user is slow or unreliable, the end result should be considered a failure.


What does performance mean to your business? Aberdeen Group surveyed 160 companies with average annual revenue of over $1 billion, finding that a one-second delay in response time caused an 11% decrease in page views, a 7% decrease in “conversions,” and a 16% decrease in customer satisfaction.

Google conducted experiments on two different page designs, one with 10 results per page and another with 30. The larger design page took a few hundred milliseconds longer to load, reducing search usage by 20%. Traffic at Google is correlated to click-through rates, and click-through rates are correlated with revenue, so the 20% reduction in traffic would have led to a 20% reduction in revenue. Reducing Google Maps’ 100-kilobyte page weight by almost a third increased traffic by over one-third.

The correlation between response time and revenue is not restricted to Google. A former employee of Amazon.com discovered that 100 milliseconds of delay reduced revenues by 1%. Whether you are selling goods online or providing access to healthcare registration for citizens, there is a direct correlation between the performance of your applications and the success of your business.

Why Is Effective Performance Engineering Necessary?

Over the years, we have all lived with the mantras “Do more with less,” and “Faster, cheaper, and better.” While some organizations have survived, many have not. Now we are faced with a different question: “How do we deliver highest quality, highest value, at the highest speed?”

In addition, organizations must focus on other key elements, like revenue, competitive advantage, customers, and brand value. Practices like Agile and DevOps have evolved and become more widely adopted. Regardless of the lifecycle, Effective Performance Engineering practices are enabling organizations to accomplish the preceding focus areas and goals for their end users. We will touch on each of these focus points next.


Effective Performance Engineering enables organizations to increase revenue in several ways. In a recent survey, 68% of respondents expected Performance Engineering practices to increase revenues by “ensuring the system can process transactions within the requisite time frame.” While all survey participants were generally in agreement regarding the tasks required, they had different expectations.

In addition to the increase in revenue, another result is reduced cost; 62% of the Performance Engineering–focused respondents felt that Performance Engineering practices should serve to avoid “unnecessary hardware acquisition costs.”

By building in performance, organizations can start optimizing applications before the first piece of code is even written, or before that new capability lights up the hardware, thereby improving the end-user experience and proactively focusing on the business objectives. Cost reductions can be dramatic. As the result of reduced performance-related production incidents, organizations can often handle 30–50 percent more transactions with the same (or less) infrastructure.

Savings can quickly multiply—fewer machines means reduced capital expense as the business scales, including lower operational expenses related to power and cooling. Fewer resources are then needed to support the infrastructure. Savings also accumulate through the reduction in performance-related production incidents that need to be managed, which reduces the opportunity cost of putting your most valuable people to work on new features and functions for the end user and your business.

A catalog company, for example, might focus on total revenue from a specific product line or service, then track it after making specific changes to a promotional website to see if revenue increases. In another scenario, the value of the mobile application might be judged by the number of registered users and the frequency with which they access products or services. And, as the backend infrastructure (including web servers, middle tiers, and databases) takes on more roles inside the corporation, the metrics that track the performance of the larger organizational goals better reflect the quality of the supporting technology.

The rise in importance of Performance Engineering has been driven by practical concerns. At least 50% of respondents admitted that slowdowns and outages were discouraging customers and frustrating employees. Many characterized the problems as “repeated,” and said they were often caused by large spikes in traffic that weren’t anticipated when the applications were built.

The consequences are serious. The average firm that responded to the survey said that a major outage could cost between $100,000 and $500,000 in lost revenue per hour. Some of the larger companies with more than 10,000 employees said they could lose $5 million an hour from website or core system outages.

When organizations contemplated the scope and catastrophic range of these failures, they recognized that the traditional development process just wasn’t ready to build a system with adequate provisions for surviving these kinds of issues. Transforming the organization and focusing individuals and teams on performance means empowering them with capabilities to anticipate problems and solve them before they occur. And when problems emerge after deployment, it means giving the team the ability to control failure and mitigate risk.

This is one of the reasons why a greater understanding of Performance Engineering as a cross-discipline, intra-business mindset is so essential. Revenue is often the first and foremost measurement of why something needs to change in any organization, but it is not always used as a measurement of Effective Performance Engineering—even though it’s often seen as the key differentiator in gaining a competitive advantage related to delivering a product or service faster and better than anyone else in the market.

Competitive Advantage

There’s an obvious business reason to focus on the needs of end users. If they’re your customers, they’re consuming your products and/or services and possibly paying you for results. If you’re a provider in a technology chain that defines a complex solution of some sort, you’re still dependent on the satisfaction of users, however indirectly.

But it wasn’t so long ago—say, 20 years—that users of computing systems were expected to live with high latencies, downtimes, and bug workarounds that were common in business systems. That’s because users were employees and had to put up with the buggy, slow, or unpredictable systems their employers provided. Or they were users of a standalone application, such as WordPerfect for writing, or Lotus 1-2-3 for spreadsheets.

That’s right, we’re talking about the pre-Internet age, when very few users imagined doing actual business transactions online. But once e-commerce became a buzzword, and soon simply a word, users stopped being just users. They became customers, and it was obvious the best web-based experiences for customers would lead to repeat business.

Fast-forward to today’s web-based and mobile business climate, where:

  • User experience (UX) is a red-hot topic.

  • Commoditization of virtually everything is a fact of life.

  • Social media is the engine that can quickly sink an online retailer, transportation provider, and so on if the UX is poor—no matter the reason.

These days, performance is king, and your online or mobile customers can either be adoring subjects or a mob of thousands, with the social-media equivalent of torches and pitchforks, at your kingdom’s gate. Or they’ll just go on to the next provider, leaving you alone in the dark ages.

Thrill your end users (especially customers) by outperforming their expectations, and you’ll get to keep doing more of whatever made them come to you in the first place, while making your company brand champions.

Many companies recognize that it’s not enough to call something a glitch, cross their virtual fingers, and hope it never happens again. They need to track their computer systems’ online status; guarantee that they’re responding to customers, partners, and employees; and measure whether they’re delivering the information promptly so no one is clicking, tapping, pounding the return key, and wondering whether they should just go somewhere else.

As corporate leaders realize the importance of their online and mobile presence and start to measure just how much business comes in through all of their channels, they’re reshaping organizations to keep information flowing quickly and accurately. This responsibility is gaining traction under the umbrella of Performance Engineering.

Proponents of this new vision believe that enterprises must build a performance-focused culture throughout their organizations—one that measures the experience of end users, both internal and external—and deliver a software and hardware experience that results in efficient performance for these users. Performance must be prioritized from the beginning of the process and be watched vigilantly after the code is deployed.

More efficient, faster systems leave employees less frustrated, something that almost certainly translates to customer satisfaction as well. The rest of the answers indicate a general awareness that the performance and throughput of computer systems are directly related to competitive advantage and the organization’s ability to retain and attract customers.

Customers: Acquisition and Retention

Acquiring and retaining customers should be the driving force in an organization, no matter the size. As we have mentioned, Performance Engineering plays a considerable role in enabling your organization to succeed in the marketplace, and success is defined in many ways for different organizations, and is not exclusive by industry or organization or product/service. But in general, success is defined by a few factors (and many times a combination), including: revenue, competitive advantage, brand value, and customers (acquisition and retention).

When your customers look across the market to discover where they can get a specific product or services, what do they see, and how do they evaluate it? Today, much of this process is accomplished in a number of near real-time and openly accessible formats, including app store reviews, feedback on websites, and even product- or service-specific feedback on your sites.

A myriad of feedback and comments are available to you regarding how your customers perceive your products and services, and what you need to focus on for your next release to deliver the features and functions they are seeking. The question becomes, “How are you capturing and what are you doing with this data today?” Using effective Performance Engineering practices, you can leverage a continuous feedback loop from production to market, provide the highest value to your customer, and increase your acquisition and retention.

Defining and knowing your customers is crucial. A recent infographic (cropped) posted by Rigor.com titled “Why Performance Matters to Your Bottom Line” illustrates how impatient online shoppers are, and the impact of a one-second delay (see Figure 1-2).

Why performance matters to your bottom line
Figure 1-2. Why performance matters to your bottom line

The key point to highlight from this infographic is that online shoppers are impatient, and the cost of a one-second delay is substantial; understanding this, and doing something about it for your business, is why we are focusing on building a Performance Engineering culture and adopting practices to improve quickly.

Of course, this leads to another important consideration: metrics. Thinking about acquisition and retention of customers involves measuring many key elements, including the feedback in the marketplace for your products and/or services. For many companies, this becomes a great starting point and one they leverage to engineer into their practices.

As organizations consider and implement metrics, they’re driven by the realization that computer system performance and throughput are directly related to competitive advantage and their ability to retain and attract customers. This focus on the customer is helpful both for acquiring new customers and retaining existing ones.

Brand Value

In many cases, businesses continue to invest in both a capability and a culture as they work to build in the practices of Performance Engineering. These practices enable them to grow faster and become more stable. Done well, Performance Engineering avoids the large-scale catastrophes like the one that hit Best Buy in 2014 (see Fortune, CNBC, CNN Money) as well as the soft failures that come when slow services frustrate employees and turn away customers. The big failures may get all the media attention, but it’s the gradual slowdowns that can be even more damaging, as they erode revenue and brand value. By the way, these are quick to be picked up in the media and amplified via social media channels, broadening and accelerating the damage to your brand.

Brand value is often something measured and tied to the stakeholders within your marketing organization(s). However, as we see the culture of performance and practices of Effective Performance Engineering, this is becoming relevant to all stakeholders (and for all the right reasons).

Keep your end users (especially customers) thrilled by outperforming their expectations and alternatives, and you’ll get to keep doing more of whatever made them come to you in the first place, while making them brand champions.

All survey respondents viewed the downside of poor performance in much the same way. Together, 66% agreed that poor performance could hurt brand loyalty and brand value. As more users interact with an organization through online and mobile channels, it only makes sense that performance would reflect directly on the brand.

In conclusion, invest in Effective Performance Engineering, or risk costly failures. You’ll be rewarded with smooth rollouts, lower overhead, and higher revenue. What if you don’t invest? Expect damaged brands, lower revenue, and lower employee morale.

Focusing on Business Need

The business needs to ensure that revenue, competitive advantage, customer acquisition and retention, and brand goals are achieved. Doing so means expanding products and service offerings and/or businesses either through organic or acquisition approaches, all of which may depend on an existing or new platform(s), so end users can consume products and services without interruption when, where, and how they want.

As a result, businesses should always be seeking to adopt Effective Performance Engineering capabilities. How they choose to do so must be led by clear and visible objectives, key metrics and measurements, and communications. If they support and participate, they will see positive results from the previously defined goals and objectives—if not, negative results will be sure to follow in one or many of these areas.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required