Continuous Delivery in the Wild

Chapter 1. Introduction

Software companies are under constant pressure to deliver features to their users faster, while simultaneously maintaining (or improving) the quality of those features. This may seem like an impossible task, but many organizations have discovered that it is in fact achievable, using the practices of Continuous Delivery.

What is Continuous Delivery?

Continuous Delivery is a set of technical practices that allow delivery teams to radically accelerate the pace at which they deliver value to their users. The core tenet of Continuous Delivery is keeping your codebase in a state where it can be shipped to production at any time. By working in this way, you can quicken the tempo of production changes, going from infrequent, big, and risky deployments to deployments that are frequent, small, and safe.

Note

The book Accelerate, by Nicole Forsgren, PhD, Jez Humble, and Gene Kim, uses rigorous scientific methods to confirm many of the benefits of Continuous Delivery, based on several years of the State Of Devops Survey. The book shows that organizations practicing Continuous Delivery have better software delivery performance, which in turn drives greater organizational performance. In other words, Continuous Delivery leads to better organizational outcomes.

Continuous Delivery in the Real World

A lot of the discussion around Continuous Delivery focuses on the cutting-edge practices advancing the state of the art. In this report, we will instead look at how a variety of organizations have implemented Continuous Delivery in the real world.

We’ll look at some of the common approaches these organizations have found helpful. We’ll also see how there are multiple ways to achieve the same goal, depending on the organizational context. My hope is that after reading this report you’ll come away with some actionable ideas for how to implement, or improve, Continuous Delivery within your own organization.

Research Methodology

To understand how people have implemented Continuous Delivery in the real world, I conducted in-depth interviews with a variety of organizations with a rapid software delivery tempo, deploying code into production at least daily. Some of these organizations have practiced something akin to Continuous Delivery from the start, while others have migrated to Continuous Delivery practices over the last few years.

Throughout the rest of the report, I’ll refer to the organizations interviewed as “participants.” The more interesting participants are described as follows:

Payment Processor

Founded in 2009, with around 1000 engineers and Service-Oriented Architecture (SOA) consisting of around 500 services.

Automotive Marketplace

Has had an online presence since the mid-1990s, with around 200 engineers and an architecture that’s migrating from monolithic web apps toward SOA, with currently around 300 services.

Online Retailer

Founded in the 1950s, with around 20 engineers working on a monolithic web application, which is just starting to move toward SOA.

Food Delivery Service

Founded in 2012, with around 30 engineers working on an SOA of 15 to 20 services.

Healthcare Provider

Founded in 2007, with around 100 engineers working on a monolithic web application.

Print and Design Service

Founded in 2004, with around 50 engineers who are partway through a migration from a monolithic web application to SOA, with currently around 50 services.

Online Realtor

Founded in 1995, with around 450 engineers working on an SOA made of up of both large monolithic systems and smaller services.

Financial Services Startup

Founded in 2007, with around 800 engineers working on an SOA with a large number of small services.

We’ll learn more about how each of these organizations has approached the challenges of Continuous Delivery throughout the report.

The Path to Production

To understand how each participant does software delivery, I asked them to describe the “path to production” for a small user-facing feature. While the organizations vary broadly in terms of architecture, industry, and organization size, there is a striking consistency in the mechanics of how each organization has implemented Continuous Delivery.

Across all the organizations that I surveyed, the path to production looks something like this (Figure 1-1):

• Engineer implements feature

• Change is reviewed and merged to master

• Change is validated via automated tests

• Change is automatically deployed to a shared integration environment

• Brief exploratory testing of change is done (if warranted)

• Change is deployed to production

• Controlled rollout of the change to users happens (if warranted)

Figure 1-1. The path to production: how a feature moves from an engineer’s keyboard into a user’s hands

The first half of this path—building the feature, merging it, and performing automated validation of the merged code—constitutes the practice of Continuous Integration. The second half—flowing the changes that make up the new feature through to production in a safe, consistent way—constitutes Continuous Delivery.

We’ll explore how different organizations implement this path to production over the course of this report.

Continuous Delivery Versus Continuous Deployment

Every organization I surveyed has an automated deployment from master¹ to a shared preproduction environment. Any change that lands on master will automatically be deployed to the preproduction environment, as long as it passes Continuous Integration validation.

This approach is taken even further at the Online Realtor and the Automotive Marketplace, where some teams automatically promote changes to their production environment, with no human intervention necessary, as long as it passes further automated validation. This is an example of Continuous Deployment—every valid change landing on master will automatically flow all the way through to production.

However, most of the organizations that I talked with avoid full-on Continuous Deployment. Instead, they institute some sort of manual gate, requiring an engineer to explicitly promote their changes into production from a preproduction environment. This wouldn’t be considered Continuous Deployment, but it is still a form of Continuous Delivery.

¹ Going forward, I’ll use “master” as a shorthand for the main development branch where a team integrates their work, since that’s the most common nomenclature in today’s git-centric world. This is sometimes referred to as “trunk” (i.e., in Trunk-Based Development).