DevOpsSec

Chapter 1. DevOpsSec: Delivering Secure Software through Continuous Delivery

Introduction

Some people see DevOps as another fad, the newest new-thing overhyped by Silicon Valley and by enterprise vendors trying to stay relevant. But others believe it is an authentically disruptive force that is radically changing the way that we design, deliver, and operate systems.

In the same way that Agile and Test-Driven Development (TDD) and Continuous Integration has changed the way that we write code and manage projects and product development, DevOps and Infrastructure as Code and Continuous Delivery is changing IT service delivery and operations. And just as Scrum and XP have replaced CMMi and Waterfall, DevOps is replacing ITIL as the preferred way to manage IT.

DevOps organizations are breaking down the organizational silos between the people who design and build systems and the people who run them—silos that were put up because of ITIL and COBIT to improve control and stability but have become an impediment when it comes to delivering value for the organization.

Instead of trying to plan and design everything upfront, DevOps organizations are running continuous experiments and using data from these experiments to drive design and process improvements.

DevOps is finding more effective ways of using the power of automation, taking advantage of new tools such as programmable configuration managers and application release automation to simplify and scale everything from design to build and deployment and operations, and taking advantage of cloud services, virtualization, and containers to spin up and run systems faster and cheaper.

Continuous Delivery and Continuous Deployment, Lean Startups and MVPs, code-driven configuration management using tools like Ansible and Chef and Puppet, NoOps in the cloud, rapid self-service system packaging and provisioning with Docker and Vagrant and Terraform are all changing how everyone in IT thinks about how to deliver and manage systems. And they’re also changing the rules of the game for online business, as DevOps leaders use these ideas to out-pace and out-innovate their competitors.

The success that DevOps leaders are achieving is compelling. According to Puppet Labs’ 2015 State of DevOps Report:

• High-performers deploy changes 30 times more often than other organizations, with lead times that are 200 times shorter.

• Their change success rate is 60 times higher.

• And when something goes wrong, they can recover from failures 168 times faster.

What do you need to do to achieve these kinds of results? And, how can you do it in a way that doesn’t compromise security or compliance, or, better, in a way that will actually improve your security posture and enforce compliance?

As someone who cares deeply about security and reliability in systems, I was very skeptical when I first heard about DevOps, and “doing the impossible 50 times a day.”¹ It was too much about how to “move fast and break things” and not enough about how to build systems that work and that could be trusted. The first success stories were from online games and social-media platforms that were worlds away from the kinds of challenges that large enterprises face or the concerns of small businesses.

I didn’t see anything new or exciting in “branching in code” and “dark launching” or developers owning their own code and being responsible for it in production. Most of this looked like a step backward to the way things were done 25 or 30 years ago, before CMMi and ITIL were put in to get control over cowboy coding and hacking in production.

But the more I looked into DevOps, past the hype, I found important, substantial new ideas and patterns that could add real value to the way that systems are built and run today:

Infrastructure as Code

Defining and managing system configuration through code that can be versioned and tested in advance using tools like Chef or Puppet dramatically increases the speed of building systems and offers massive efficiencies at scale. This approach to configuration management also provides powerful advantages for security: full visibility into configuration details, control over configuration drift and elimination of one-off snowflakes, and a way to define and automatically enforce security policies at runtime.

Continuous Delivery

Using Continuous Integration and test automation to build pipelines from development to test and then to production provides an engine to drive change and at the same time a key control structure for compliance and security, as well as a safe and fast path for patching in response to threats.

Continuous monitoring and measurement

This involves creating feedback loops from production back to engineering, collecting metrics, and making them visible to everyone to understand how the system is actually used and using this data to learn and improve. You can extend this to security to provide insight into security threats and enable “Attack-Driven Defense.”

Learning from failure

Recognizing that failures can and will happen, using them as learning opportunities to improve in fundamental ways through blameless postmortems, injecting failure through chaos engineering, and practicing for failure in game days; all of this leads to more resilient systems and more resilient organizations, and through Red Teaming to a more secure system and a proven incident response capability.

About DevOps

This paper is written for security analysts, security engineers, pen testers, and their managers who want to understand how to make security work in DevOps. But it also can be used by DevOps engineers and developers and testers and their managers who want to understand the same thing.

You should have a basic understanding of application and infrastructure security as well as some familiarity with DevOps and Agile development practices and tools, including Continuous Integration and Continuous Delivery. There are several resources to help you with this. Some good places to start:

• The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford is a good introduction to the hows and whys of DevOps, and is surprisingly fun to read.

• Watch “10+ Deploys per Day,” John Allspaw and Paul Hammond’s presentation on Continuous Deployment, which introduced a lot of the world to DevOps ideas back in 2009.²

• And, if you want to understand how to build your own Continuous Delivery pipeline, read Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and Dave Farley.

The more I talked to people at organizations like Etsy and Netflix who have had real success with DevOps at scale, and the more I looked into how enterprises like ING and Nordstrom and Capital One and Inuit are successfully adopting DevOps, the more I started to buy in.

And the more success that we have had in my own organization with DevOps ideas and tools and practices, the more I have come to understand how DevOps, when done right, can be used to deliver and run systems in a secure and reliable way.

Whether you call it SecDevOps, DevSecOps, DevOpsSec, or Rugged DevOps, this is what this paper will explore.

We’ll begin by looking at the security and compliance challenges that DevOps presents. Then, we’ll cover the main ideas in secure DevOps and how to implement key DevOps practices and workflows like Continuous Delivery and Infrastructure as Code to design, build, deploy, and run secure systems. In Chapter 4, we’ll map security checks and controls into these workflows. Because DevOps relies so heavily on automation, we’ll look at different tools that you can use along the way, emphasizing open source tools where they meet the need, and other tools that I’ve worked with and know well or that are new and worth knowing about. And, finally, we’ll explore how to build compliance into DevOps, or DevOps into compliance.

¹ From an early post on Continuous Deployment: http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/

² Velocity 2009: “10+ Deploys per Day.” https://www.youtube.com/watch?v=LdOe18KhtT4