Optimism is an occupational hazard of programming.
The practice of software development has been plagued with shockingly low success rates for decades. At the same time, the number of software-driven products and services continues to increase dramatically each year. If these were the only two trends, we would be heading for disaster.
Fortunately, Agile software development methods have been demonstrating that higher success rates are possible. And Lean techniques (which have dramatically increased manufacturing success for over 50 years) are now being applied to software development and validating the successes of Agile.
The Lean principles and the mindset of Lean thinking have proved remarkably applicable to improving the productivity and quality of just about any endeavor. Lean has been successfully applied to manufacturing, distribution, supply chain, product development, banking, engineering, the back office, and much more. However, only in the last few years have the Lean principles and techniques been applied to software development.
In this chapter we will give you more details on the problems that continue to plague software development, as well as an overview of Agile software development and the origins of Lean and its unique approach to improving any kind of process.
Have you ever worked on a software development project that:
Went over schedule?
Went over budget?
Didn’t meet the needs of the customer?
Was canceled?
If you said no, you are probably fresh out of school and are still working on your first project. If you said yes, you’re not alone…not by a long shot!
The actual statistics are shocking.
The 1994 CHAOS Report from the Standish Group is the landmark study of IT project failure. By 1994, the Standish Group had studied over 8,000 software development projects and found that only 16% were successful. This means that 84% of the projects either failed outright or had serious problems. In 2004, after 10 years of study, the number of projects included had increased to 40,000, and the success rate had improved to 29%. Although this is a significant increase, it’s still nothing to brag about.
Can you think of any other industry that has such a staggeringly low success rate?
The Standish Group survey included large, medium, and small companies across major industry segments: banking, securities, manufacturing, retail, wholesale, health care, insurance, services, and local, state, and federal organizations (365 companies in total). Over a 10-year period, it studied over 40,000 projects. In addition, it used focus groups and personal interviews to provide a qualitative context for the survey results.
The CHAOS study classified projects into three categories:
- Project Succeeded
The project was completed on time and on budget with all the features and functions that were originally specified.
- Project Challenged
The project was completed, but went over budget, over the time estimate, and offered fewer features and functions than originally specified.
- Project Failed
The project was canceled at some point during the development cycle (the study actually called this Project Impaired).
Figure 1-1 shows the percent of projects in each of these categories over a 10-year period.
As you can see, the success rate improved over time, but at a glacial pace: 1.3% a year. At this rate, successful projects won’t even break the 50% mark until the year 2020!
In fairness, some experts disagree with the definition of success used by the CHAOS study. They point out that many projects that have been classified as challenged have gone on to deliver very successful products. However, even if you take this into account, the success rate still leaves a lot of room for improvement.
So the question is: why is the failure rate so high? A large part of the blame can be traced back to the widespread adoption of the Waterfall method.
The Waterfall model of software development divides development into a set of distinct phases that are performed sequentially: requirements, design, implementation, testing, deployment, and maintenance (see Figure 1-2). Development is seen as flowing steadily downward (like a waterfall) through these phases. Each phase has a beginning and an end, and once you move onto the next phase, you never go back (just as water does not flow uphill).
The Waterfall method has an appealing air of simplicity. Managers like the series of fixed milestones that seem to make it easy to track a project’s progress. However, this manageability is an illusion because this model does not allow for change.
Software development is a highly dynamic process and change is inevitable. During implementation you’ll find problems with the design. Customers don’t know precisely what they want ahead of time, and that will change the requirements.
Studies show that the Waterfall method is positively correlated with reduced productivity, increased defect rates, and project failure. Why, then, was the Waterfall method so widely promoted, and how did it become so entrenched in the face of so much evidence that it did not work?
Winston Royce first described the Waterfall method in a 1970 paper titled, “Managing the Development of Large Software Systems” (Technical Papers of the Western Electronic Show and Convention). This paper is often cited as if it validates the Waterfall model, but it actually does the opposite. People who use Royce’s paper to support the Waterfall method must not have read this paper carefully because it explicitly says that the Waterfall method “is risky and invites failure.” The paper then proceeds to advocate an iterative development style.
The Waterfall method would likely have slowly faded away, but in the 1980s it became the Department of Defense (DoD) standard for the development and procurement of software with the release of DOD-STD-2167. Eventually, the DoD realized that the Waterfall method was not working and in 1994 replaced DOD-STD-2167 with MIL-STD-498, which supports iterative development.
However, the damage was done, and a strong mindset with a bias toward Waterfall development had become ingrained. The Lightweight methods of the 1990s and the Agile methods of the early 2000s have started to turn this around, but there is a long way to go. Many people who are not familiar with the evidence are still staunch supporters of the Waterfall method.
Get The Art of Lean Software Development now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
