Optimism is an occupational hazard of programming.
The practice of software development has been plagued with shockingly low success rates for decades. At the same time, the number of software-driven products and services continues to increase dramatically each year. If these were the only two trends, we would be heading for disaster.
Fortunately, Agile software development methods have been demonstrating that higher success rates are possible. And Lean techniques (which have dramatically increased manufacturing success for over 50 years) are now being applied to software development and validating the successes of Agile.
The Lean principles and the mindset of Lean thinking have proved remarkably applicable to improving the productivity and quality of just about any endeavor. Lean has been successfully applied to manufacturing, distribution, supply chain, product development, banking, engineering, the back office, and much more. However, only in the last few years have the Lean principles and techniques been applied to software development.
In this chapter we will give you more details on the problems that continue to plague software development, as well as an overview of Agile software development and the origins of Lean and its unique approach to improving any kind of process.
Have you ever worked on a software development project that:
Went over schedule?
Went over budget?
Didn’t meet the needs of the customer?
If you said no, you are probably fresh out of school and are still working on your first project. If you said yes, you’re not alone…not by a long shot!
The actual statistics are shocking.
The 1994 CHAOS Report from the Standish Group is the landmark study of IT project failure. By 1994, the Standish Group had studied over 8,000 software development projects and found that only 16% were successful. This means that 84% of the projects either failed outright or had serious problems. In 2004, after 10 years of study, the number of projects included had increased to 40,000, and the success rate had improved to 29%. Although this is a significant increase, it’s still nothing to brag about.
Can you think of any other industry that has such a staggeringly low success rate?
The Standish Group survey included large, medium, and small companies across major industry segments: banking, securities, manufacturing, retail, wholesale, health care, insurance, services, and local, state, and federal organizations (365 companies in total). Over a 10-year period, it studied over 40,000 projects. In addition, it used focus groups and personal interviews to provide a qualitative context for the survey results.
The CHAOS study classified projects into three categories:
The project was completed on time and on budget with all the features and functions that were originally specified.
The project was completed, but went over budget, over the time estimate, and offered fewer features and functions than originally specified.
The project was canceled at some point during the development cycle (the study actually called this Project Impaired).
Figure 1-1 shows the percent of projects in each of these categories over a 10-year period.
As you can see, the success rate improved over time, but at a glacial pace: 1.3% a year. At this rate, successful projects won’t even break the 50% mark until the year 2020!
In fairness, some experts disagree with the definition of success used by the CHAOS study. They point out that many projects that have been classified as challenged have gone on to deliver very successful products. However, even if you take this into account, the success rate still leaves a lot of room for improvement.
The Waterfall model of software development divides development into a set of distinct phases that are performed sequentially: requirements, design, implementation, testing, deployment, and maintenance (see Figure 1-2). Development is seen as flowing steadily downward (like a waterfall) through these phases. Each phase has a beginning and an end, and once you move onto the next phase, you never go back (just as water does not flow uphill).
The Waterfall method has an appealing air of simplicity. Managers like the series of fixed milestones that seem to make it easy to track a project’s progress. However, this manageability is an illusion because this model does not allow for change.
Software development is a highly dynamic process and change is inevitable. During implementation you’ll find problems with the design. Customers don’t know precisely what they want ahead of time, and that will change the requirements.
Studies show that the Waterfall method is positively correlated with reduced productivity, increased defect rates, and project failure. Why, then, was the Waterfall method so widely promoted, and how did it become so entrenched in the face of so much evidence that it did not work?
Winston Royce first described the Waterfall method in a 1970 paper titled, “Managing the Development of Large Software Systems” (Technical Papers of the Western Electronic Show and Convention). This paper is often cited as if it validates the Waterfall model, but it actually does the opposite. People who use Royce’s paper to support the Waterfall method must not have read this paper carefully because it explicitly says that the Waterfall method “is risky and invites failure.” The paper then proceeds to advocate an iterative development style.
The Waterfall method would likely have slowly faded away, but in the 1980s it became the Department of Defense (DoD) standard for the development and procurement of software with the release of DOD-STD-2167. Eventually, the DoD realized that the Waterfall method was not working and in 1994 replaced DOD-STD-2167 with MIL-STD-498, which supports iterative development.
However, the damage was done, and a strong mindset with a bias toward Waterfall development had become ingrained. The Lightweight methods of the 1990s and the Agile methods of the early 2000s have started to turn this around, but there is a long way to go. Many people who are not familiar with the evidence are still staunch supporters of the Waterfall method.
Most of the Agile methodologies, like Extreme Programming (XP) and Scrum, actually predate the Agile Manifesto (which we will discuss shortly). In fact, using the term “Agile” to refer to these methods of software development was coined at the famous Snowbird meeting that created the Agile Manifesto.
In the 1990s there was a growing dissatisfaction with the prevalent heavy software development methodologies and processes. Using these processes did not solve any of the endemic problems of software development: high project failure rate, low software quality, and generally unhappy customers.
This spawned a number of alternative methodologies, including XP, Scrum, Dynamic Systems Development Method (DSDM), Crystal, and Feature Driven Development (FDD), that were collectively known as Lightweight methods. The term “Lightweight” was meant to distinguish them from the predominant heavyweight methods of the time. Their creators were not happy with the term “Lightweight” because it seemed to imply that these methods were less comprehensive or less important.
It became increasingly apparent that these Lightweight methods had a lot in common with each other. So, in February of 2001, a group of 17 of the leading independent thinkers about software development gathered at the Snowbird ski resort in Utah to try to find common ground. The roster of this two-day gathering included many of the most prominent personalities in software development at the time: Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, and Dave Thomas.
Three significant things came out of the now-famous Snowbird gathering: the decision to use the term “Agile,” the Agile Manifesto, and the Agile Alliance. The Agile Alliance (http://www.agilealliance.org/) is a nonprofit organization that exists to further the development and dissemination of information regarding Agile processes.
As stated earlier, no one was happy with the term “Lightweight.” Alistair Cockburn articulated this nicely when he said, “I don’t mind the methodology being called light in weight, but I’m not sure I want to be referred to as a ‘Lightweight’ attending a ‘Lightweight methodologists’ meeting. It sounds like a bunch of skinny, feebleminded people trying to remember what day it is.” And thus, we now have “Agile.”
The Agile Manifesto (formally called the Manifesto for Agile Software Development) is a model of simplicity. Written by Kent Beck et al., it eloquently states the four core values that form the philosophical bedrock of all Agile methodologies:
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
The wording of the Agile Manifesto was carefully crafted to communicate the intended message. By saying, “We are uncovering better ways of developing software,” they showed that they didn’t have all the answers and were still on a learning journey of discovery. Just as important was the next part, “by doing it and helping others do it,” which meant that they were actively engaged in using these “better ways” (no ivory-tower theoretical advice here) and were sharing what they learned with others (not telling people what to do).
Each of the four value statements in the Agile Manifesto lists two things and states that both items are important, but the first item is higher priority. Martin Fowler and Jim Highsmith described this very succinctly in their August 2001 article in Dr. Dobb’s Journal titled “The Agile Manifesto”:
The Alliance recognizes the importance of process and tools, with the additional recognition that the interaction of skilled individuals is of even greater importance. Similarly, comprehensive documentation is not necessarily bad, but the primary focus must remain on the final product—delivering working software. Therefore, every project team needs to determine for itself what documentation is absolutely essential.
Contract negotiation, whether through an internal project charter or external legal contract, isn’t a bad practice, just an insufficient one. Contracts and project charters may provide some boundary conditions within which the parties can work, but only through ongoing collaboration can a development team hope to understand and deliver what the client wants.
Plans are important, but they shouldn’t be rigid, unchanging plans. The ability to respond to changes is critical to the success of most software development projects.
In addition to enumerating the four Agile values, the authors of the Agile Manifesto further refined what they meant in those value statements by listing a number of principles that they follow as a direct result of those values:
Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity—the art of maximizing the amount of work not done—is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
There are a fair number of formal software development methodologies that fall within the Agile camp. While they vary in their specific activities and artifacts, they all use short time-boxed iterations that deliver working software at the end of each iteration (the length of the iterations varies between methodologies). Figure 1-3 shows the typical Agile process.
All Agile methodologies share a number of core practices (either formally or informally). These core practices also support Lean software development and are the subject of the second half of this book.
Some of the more commonly used Agile methodologies include:
Unified Process (UP)
These methods all began at different times in the 1990s as a response to the failure of the Waterfall method. There has been a great deal of cross-fertilization of ideas and techniques between these methods.
Scrum has successfully taken the Agile approach and stripped it down to the essentials. The result is one of the simplest and easiest-to-implement Agile methodologies that still provides the benefits of Agile software development.
XP has a rich set of interlocking practices that can feel overwhelming to those uninitiated in the Agile way, but XP gets credit for popularizing most of the core practices that have been adopted by the other methodologies. XP has probably done more to raise the awareness of Agile software development than any other methodology.
Crystal is actually a family of methodologies created by Alistair Cockburn. The actual processes and practices vary depending on a project’s size and criticality or complexity.
FDD is unique among the Agile methodologies in that its perspective centers on creating domain models for the system being developed, which then organizes development around features that implement the model.
UP is generally considered to be one of the more “heavyweight” of the Agile processes, although it was intended to be tailored and not implemented in a heavy manner. Regardless, this has led to a number of variants, including the Rational Unified Process (RUP), the Agile Unified Process (AUP), and the Enterprise Unified Process (EUP).
DSDM is more formal than most of the other Agile methods, fully specifying many different roles, processes, and artifacts. A notable feature is the prioritizing of requirements according to the MoSCoW rules:
It’s only the last turn of a bolt that tightens it—the rest is just movement.
It is only recently that the Lean principles have been applied to software development. In the beginning it all started with Lean manufacturing (some 40 to 60 years ago, depending on when you start the clock). Of course, it wasn’t called Lean back then. It was the Toyota Production System, or Just-In-Time manufacturing. James Womack, Daniel Jones, and Daniel Roos coined the term “Lean” in their 1990 book, The Machine That Changed the World (Harper Perennial).
Lean is a mindset, a way of thinking about how to deliver value to the customer more quickly by finding and eliminating waste (the impediments to quality and productivity). This philosophy is expressed in a set of principles that have proven remarkably applicable to a wide range of business activities. The Lean principles have been used successfully in areas as diverse as the supply chain, the office, engineering, and (of course) software development.
First, we’ll take a look at the roots of Lean thinking with a short history of how Lean developed in manufacturing. In Chapter 2, we’ll show you how you can apply Lean thinking to software development and how it differs from Agile.
To really appreciate the emergence of Lean production and its derivatives, you have to understand what it was replacing (or competing with): mass production.
Henry Ford popularized mass production (which had itself replaced craft production) with the assembly-line manufacture of the Model T in 1913. Mass production is used to produce large quantities of goods at a low unit cost. It divides the manufacturing process into small steps that can be carried out by unskilled workers, and it relies on the use of high-precision machinery and standardized, interchangeable parts.
The drawback of mass production is its inflexibility. Since a mass production assembly line is so expensive to set up (and difficult to alter), it is only economical if it is going to produce large quantities of the same thing.
In 1945, in post-war Japan, the president of Toyota Motor Company, Kiichiro Toyoda, said that the Japanese automobile industry would not survive if it did not “catch up with America in three years.” This did not happen within three years (the Japanese automobile industry survived anyway), but it did lead to the creation of the Toyota Production System by Taiichi Ohno.
Taiichi Ohno realized that the American mass production system would not work in Japan. The domestic automobile market was too small and there was a significant demand for variety in automobiles, from small economical cars to larger, luxury cars—a poor fit for mass production. Out of necessity, Taiichi Ohno experimented with many ideas and techniques that eventually evolved into the Toyota Production System.
Taiichi Ohno described the Toyota Production System as “a system for the absolute elimination of waste.” He wasn’t kidding. By the early 1990s, Toyota was 60% more productive with 50% fewer defects than its non-Lean competitors. According to Ohno, this striking advantage rested on two pillars: Just-In-Time and autonomation.
In the 1950s, a Japanese delegation from Toyota visited American businesses to study their methods. They visited Ford Motor Company, the industry leader, but they were not impressed. They were particularly appalled by the large amounts of inventory and the unevenness of the amount of work performed in different parts of the factory.
However, when they visited an American supermarket, they were impressed with the way in which products were reordered and restocked only after customers purchased them. This pull system inspired Taiichi Ohno to create Just-In-Time, which strives to keep inventories at each manufacturing step as low as possible (preferably zero). It is about providing the right material, in the right amount, at the right time, and in the right place.
According to Ohno, inventory is waste that costs the company money. Even worse, inventory hides problems in the production system. This includes problems such as inadequate capacity, inflexible equipment, and unreliable equipment. A major contribution of a Just-In-Time system is that it exposes the causes of inventory-keeping so that they can be addressed.
Autonomation is a combination of the words autonomous and automation. It describes machines that automate a process but are also intelligent enough to know when something is wrong and stop immediately. This kind of machine can run unattended (autonomously) while providing workers with full confidence that it is operating flawlessly. It doesn’t have to be monitored, and it needs human attention only when it stops.
When it detects an abnormal condition, the machine will stop itself and a worker will halt the production line. This focuses everyone’s attention on finding the root cause of the problem and fixing it so that it will not recur. In this way, autonomation prevents the production of defective components that would otherwise disrupt the production line or result in more costly rework at a later stage.
The overarching goal of Lean production (or any form of Lean) is to deliver value to the customer more quickly, and the primary way to do this is to find and eliminate waste. On the surface this may seem like a simple thing, but exactly what is and what is not waste isn’t always obvious. Shigeo Shingo, who codeveloped the Toyota Production System with Taiichi Ohno, identified seven kinds of waste (described in the DOTWIMP sidebar).
A key Lean activity is to break down a process into a map of its individual steps and identify which steps add value and which steps do not—the waste. This is known as a value stream map. The goal, then, is to eliminate the waste (muda) and improve the value-added steps (kaizen). The waste is further subdivided into two categories: non-value-added but necessary (given the current state of the system), and pure waste. Pure waste is easy to deal with—it can be eliminated immediately.
So, there are two key Lean skills: knowing what the customer values, and knowing how to spot waste.
Taiichi Ohno started with Just-In-Time and autonomation, the two pillars of the Toyota Production System. Modern-day Lean has settled on five principles and a wide array of practices that have been distilled from the Toyota Production System and the experiences of other companies that have followed Toyota’s lead. These five principles are identified as Value, Value Stream, Flow, Pull, and Perfection:
Once you know what the customer values in your product, you can create a value stream map that identifies the series of steps required to produce the product. Each step is categorized as either value-added, non-value-added but necessary, or non-value-added waste.
In the next chapter, we will see how these Lean principles can be applied to software development.