Continuous Delivery with Spinnaker

Chapter 1. Why Continuous Delivery?

Continuous delivery is the practice by which software changes can be deployed to production in a fast, safe, and automatic way.

In the continuous delivery world, releasing new functionality is not a world-shattering event where everyone in the company stops working for weeks following a code freeze and waits nervously around dashboards during the fateful minutes of deployment. Instead, releasing new software to users should be routine, boring, and so easy that it can happen many times a day.

In this chapter, we’ll describe the organizational and technical practices that enable continuous delivery. We hope that it convinces you of the benefits of a shorter release cycle and helps you understand the culture and practices that inform the delivery culture at Netflix and other similar organizations.

The Problem with Long Release Cycles

Dependencies drift. As undeployed code sits longer and longer, the libraries and services it depends upon move on. When it does come time to deploy those changes, unexpected issues will arise because library versions upstream have changed, or a service it talks to no longer has that compatible API.

People also move on. Once a feature has finished development, developers will naturally gravitate to the next project or set of features to work on. Information is no longer fresh in the minds of the creators, so if a problem does arise, they need to go back and investigate ideas from a month, six months, or a year ago. Also, by having large releases, it becomes much more difficult to isolate and triage the source of issues.

So how do we make this easier? We release more often.

Benefits of Continuous Delivery

Continuous delivery removes the ceremony around the software release process. There are several benefits to this approach:

Innovation

Continuous delivery ensures quicker time to market for new features, configuration changes, experiments, and bug fixes. An aggressive release cadence ensures that broken things get fixed quickly and new ways to delight users arrive in days, not months.

Faster feedback loops

Smaller changes deployed frequently makes it easier to troubleshoot issues. By incorporating automated testing techniques like chaos engineering or automated canary analysis into the delivery process, problems can be detected more quickly and fixed more effectively.

Increase reliability and availability

To release quickly, continuous delivery encourages tooling to replace manual error-prone processes with automated workflows. Continuous delivery pipelines can further be crafted to incrementally roll out changes at specific times and different cloud targets. Safe deployment practices can be built into the release process and reduce the blast radius of a bad deployment.

Developer productivity and efficiency

A more frequent release cadence helps reduce issues such as incompatible upstream dependencies. Accelerating the time between commit and deploy allows developers to diagnose and react to issues while the change is fresh in their minds. As developers become responsible for maintaining the services they deploy, there is a greater sense of ownership and less blame game when issues do arise. Continuous delivery leads to high performing, happier developers.

Useful Practices

As systems evolve and changes are pushed, bugs and incompatibilities can be introduced that affect the availability of a system. The only way to enable more frequent changes is to invest in supporting people with better tooling, practices, and culture.

Here are some useful techniques and principles we’ve found that accelerate the adoption of continuous delivery practices:

Encourage self-sufficiency

Instead of delegating the deployment process to a specialized team, allow the engineers who wrote the code to be responsible for deploying and supporting their own releases. By providing self-serve tools and empowering engineers to push code when they feel it is ready, engineers can quickly innovate, detect, and respond.

Automate all the things

Fully embracing automation at every step in the build, test, release, promote cycle reduces the need to babysit the deployment process.

Make it visible

It is difficult to improve things that cannot be observed. We found that consolidating all the cloud resources across different accounts, regions, and cloud providers into one view made it much easier to track and debug any infrastructure issues. Deployment pipelines also allowed our users to easily follow how an artifact was being promoted across different steps.

Make it easy to do

It shouldn’t require expert-level knowledge to craft a cloud deployment. We found that focusing heavily on user experience so that anyone can modify and improve their own processes had a significant impact in adopting continuous delivery.

Paved road

It is much easier to convince a team to embrace continuous delivery when you provide them with a ready-made template they can plug into. We defined a “paved road” (sometimes called a “golden road”) that encapsulates best practices for teams wishing to deploy to the cloud (Figure 1-1). As more and more teams started using the tools, any improvements we made as part of the feedback loop became readily available for other teams to use. Best practices can become contagious.

Figure 1-1. The paved road of software release at Netflix. The top row shows the steps, from code check-in to taking traffic, and the bottom rows show the tools used at Netflix for each step.

Summary

After migrating to a continuous delivery platform, we found the number of issues and outages caused by bad deployments reduced significantly. Now that we are all-in on Spinnaker, it is even easier to help push these practices further, resulting in a widespread reduction in deployment-related issues.