DevOpsSec

Chapter 3. Keys to Injecting Security into DevOps

Now let’s look at how to solve these problems and challenges, and how you can wire security and compliance into DevOps.

Building Security into DevOps: Etsy’s Story

Etsy, a successful online crafts marketplace, is famous for its Continuous Deployment model, where engineers (and managers and even pets) push changes out 50 times or more every day. It is also known for its blameless, “Just Culture,” in which engineers are taught to embrace failure, as long as they learn from their mistakes.

Etsy’s security culture is built on top of its engineering culture and connects with the wider culture of the organization. Some of its driving principles are:

• Trust people to do the right thing, but still verify. Rely on code reviews and testing and secure defaults and training to prevent or catch mistakes. And if mistakes happen in production, run postmortems to examine and understand what happened and how to fix things at the source.

• “If it Moves, Graph it.” Make data visible to everyone so that everyone can understand and act on it, including information about security risks, threats, and attacks.

• “Just Ship It.” Every engineer can push to production at any time. This includes security engineers. If something is broken and you can fix it, fix it and ship the fix out right away. Security engineers don’t throw problems over the wall to dev or ops if they don’t have to. They work with other teams to understand problems and get them fixed, or fix the problem themselves if they can. Everyone uses the Continuous Deployment pipelines and the same tools to push changes out to production, including the security team.

• Security cannot be a blocker. The word “No” is a finite resource—use it only when you must. Security’s job is to work with development and operations to help them to deliver, but deliver safely. This requires security to be practical and make realistic trade-offs when it comes to security findings. Is this problem serious enough that it needs to block code from shipping now? Can it be fixed later? Or, does it really need to be fixed at all? Understand the real risk to the system and to the organization and deal with problems appropriately. By not crying wolf, the security team knows that when serious problems do come up, they will be taken seriously by everyone.

Etsy’s security team takes a number of steps to build relationships between the security team and engineering.

“Designated Hackers” is a system by which each security engineer supports four or five development teams across the organization and are involved in design and standups. The security engineer tries to understand what these teams are doing and raise a signal if a security risk or question comes up that needs to be resolved. They act as a channel and advocate between security and the development teams. This helps to build relationships, and builds visibility into design and early stage decisions—when security matters most.

Every new engineering hire participates in one-week boot camps where they can choose to work with the security team to understand what they do and help to solve problems. And each year every engineer does a senior rotation where they spend a month with another team and can choose to work with the security team. These initiatives build understanding and relationships between organizations and seed security champions in engineering.¹

Shift Security Left

To keep up with the pace of Continuous Delivery, security must “shift left,” earlier into design and coding and into the automated test cycles, instead of waiting until the system is designed and built and then trying to fit some security checks just before release. In DevOps, security must fit into the way that engineers think and work: more iterative and incremental, and automated in ways that are efficient, repeatable, and easy to use. See Figure 3-1 for a comparison between waterfall delivery and the DevOps cycle.

Figure 3-1. The waterfall cycle versus the DevOps cycle

Some organizations do this by embedding infosec specialists into development and operations teams. But it is difficult to scale this way. There are too few infosec engineers to go around, especially ones who can work at the design and code level. This means that developers and operations need to be given more responsibility for security, training in security principles and practices, and tools to help them build and run secure systems.

Developers need to learn how to identify and mitigate security risks in design through threat modeling (looking at holes or weaknesses in the design from an attacker’s perspective), and how to take advantage of security features in their application frameworks and security libraries to prevent common security vulnerabilities like injection attacks. The OWASP and SAFECode communities provide a lot of useful, free tools and frameworks and guidance to help developers with understanding and solving common application security problems in any kind of system.

OWASP Proactive Controls

The OWASP Proactive Controls is a set of secure development practices, guidelines, and techniques that you should follow to build secure applications. These practices will help you to shift security earlier into design, coding, and testing:

Verify for security early and often

Instead of leaving security testing and checks to the end of a project, include security in automated testing in Continuous Integration and Continuous Delivery.

Parameterize queries

Prevent SQL injection by using a parameterized database interface.

Encode data

Protect the application from XSS and other injection attacks by safely encoding data before passing it on to an interpreter.

Validate all inputs

Treat all data as untrusted. Validate parameters and data elements using white listing techniques. Get to know and love regex.

Implement identity and authentication controls

Use safe, standard methods and patterns for authentication and identity management. Take advantage of OWASP’s Cheat Sheets for authentication, session management, password storage, and forgotten passwords if you need to build these functions in on your own.

Implement appropriate access controls

Follow a few simple rules when implementing access control filters. Deny access by default. Implement access control in a central filter library—don’t hardcode access control checks throughout the application. And remember to code to the activity instead of to the role.

Protect data

Understand how to use crypto properly to encrypt data at rest and in transit. Use proven encryption libraries like Google’s KeyCzar and Bouncy Castle.

Implement logging and intrusion detection

Design your logging strategy with intrusion detection and forensics in mind. Instrument key points in your application and make logs safe and consistent.

Take advantage of security frameworks and libraries

Take advantage of the security features of your application framework where possible, and fill in with special-purpose security libraries like Apache Shiro or Spring Security where you need to.

Error and exception handling

Pay attention to error handling and exception handling throughout your application. Missed error handling can lead to runtime problems, including catastrophic failures. Leaking information in error handling can provide clues to attackers; don’t make their job easier than it already is.

Secure by Default

Shifting Security Left begins by making it easy for engineers to write secure code and difficult for them to make dangerous mistakes, wiring secure defaults into their templates and frameworks, and building in the proactive controls listed previously. You can prevent SQL injection at the framework level by using parameterized queries, hide or simplify the output encoding work needed to protect applications from XSS attacks, enforce safe HTTP headers, and provide simple and secure authentication functions. You can do all of this in ways that are practically invisible to the developers using the framework.

Making software and software development secure by default is core to the security programs at organizations like Google, Facebook, Etsy, and Netflix. Although the pay back can be huge, it demands a fundamental change in the way that infosec and development work together. It requires close collaboration between security engineers and software engineers, strong application security knowledge and software design, and coding skills to build security protection into frameworks and templates in ways that are safe and easy to use. Most important, it requires a commitment from developers and their managers to use these frameworks wherever possible.

Most of us aren’t going to be able to start our application security programs here; instead, we’ll need to work our way back to the beginning and build more security into later stages.

Making Security Self-Service

Engineers in DevOps shops work in a self-service environment. Automated Continuous Integration servers provide self-service builds and testing. Cloud platforms and virtualization technologies like Vagrant provide self-service, on-demand provisioning. Container technologies like Docker offer simple, self-service packaging and shipping.

Security needs to be made available to the team in the same way: convenient, available when your engineers need it, seamless, and efficient.

Don’t get in their way. Don’t make developers wait for answers or stop work in order to get help. Give them security tools that they can use and understand and that they can provision and run themselves. And ensure that those tools fit into how they work: into Continuous Integration and Continuous Delivery, into their IDEs as they enter code, into code pull requests. In other words, ensure that tests and checks provide fast, clear feedback.

At Twitter, security checks are run automatically every time a developer makes a change. Tools like Brakeman check the code for security vulnerabilities, and provide feedback directly to the developer if something is found, explaining what is wrong, why it is wrong, and how to fix it. Developers have a “bullshit” button to reject false positive findings. Security checks become just another part of their coding cycle.²

Using Infrastructure as Code

Another key to DevOps is Infrastructure as Code: managing infrastructure configuration in code using tools like Ansible, Chef, Puppet, and Salt. This speeds up and simplifies provisioning and configuring systems, especially at scale. It also helps to ensure consistency between systems and between test and production, standardizing configurations and minimizing configuration drift, and reduces the chance of an attacker finding and exploiting a one-off mistake.

Treating infrastructure provisioning and configuration as a software problem and following software engineering practices like version control, peer code reviews, refactoring, static analysis, automated unit testing and Continuous Integration, and validating and deploying changes through the same kind of Continuous Delivery pipelines makes configuration changes repeatable, auditable, and safer.

Security teams also can use Infrastructure as Code to program security policies directly into configuration, continuously audit and enforce policies at runtime, and to patch vulnerabilities quickly and safely.

UpGuard: From Infrastructure to Secure Code

UpGuard is a service that helps you to automatically capture configuration information into tests and code, and then establish policies and monitor compliance against those policies. UpGuard discovers configuration details from Linux and Windows servers, network devices and cloud services, and tracks changes to this information over time. You can use it as a Tripwire-like detective change control tool to alert you to unauthorized changes to configuration or to audit configuration management activities.

You can also visualize the configuration of your systems and identify inconsistencies between them. And you can establish policies for different systems or types of systems by creating fine-grained automated tests or assertions to ensure that the proper versions of packages are installed, that specific files and users and directories are set up correctly, that ports are opened or closed, and that processes are running.

UpGuard automatically creates directives for configuration management tools, including Ansible, Chef, Puppet, Microsoft Windows PowerShell DSC, and Docker, to bring your infrastructure configuration into code with a prebuilt test framework.

It continuously assesses the risk of your configuration, assigning a score based on compliance (test coverage and pass ratios as well as compliance with external benchmarks like CIS), integrity (tracking of unauthorized changes to configuration), and security (based on scanning for vulnerabilities—UpGuard includes a community-based vulnerability scanner and can integrate with other scanners).

Iterative, Incremental Change to Contain Risks

DevOps and Continuous Delivery reduce the risk of change by making many small, incremental changes instead of a few “big bang” changes.

Changing more often exercises and proves out your ability to test and successfully push out changes, enhancing your confidence in your build and release processes. Additionally, it forces you to automate and streamline these processes, including configuration management and testing and deployment, which makes them more repeatable, reliable, and auditable.

Smaller, incremental changes are safer by nature. Because the scope of any change is small and generally isolated, the “blast radius” of each change is contained. Mistakes are also easier to catch because incremental changes made in small batches are easier to review and understand upfront, and require less testing.

When something does go wrong, it is also easier to understand what happened and to fix it, either by rolling back the change or pushing a fix out quickly using the Continuous Delivery pipeline.

It’s also important to understand that in DevOps many changes are rolled out dark. That is, they are disabled by default, using runtime “feature flags” or “feature toggles”. These features are usually switched on only for a small number of users at a time or for a short period, and in some cases only after an “operability review” or premortem review to make sure that the team understands what to watch for and is prepared for problems.³

Another way to minimize the risk of change in Continuous Delivery or Continuous Deployment is canary releasing. Changes can be rolled out to a single node first, and automatically checked to ensure that there are no errors or negative trends in key metrics (for example, conversion rates), based on “the canary in a coal mine” metaphor. If problems are found with the canary system, the change is rolled back, the deployment is canceled, and the pipeline shut down until a fix is ready to go out. After a specified period of time, if the canary is still healthy, the changes are rolled out to more servers, and then eventually to the entire environment.

Use the Speed of Continuous Delivery to Your Advantage

The speed at which DevOps moves can seem scary to infosec analysts and auditors. But security can take advantage of the speed of delivery to respond quickly to security threats and deal with vulnerabilities.

A major problem that almost all organizations face is that even when they know that they have a serious security vulnerability in a system, they can’t get the fix out fast enough to stop attackers from exploiting the vulnerability.

The longer vulnerabilities are exposed, the more likely the system will be, or has already been, attacked. WhiteHat Security, which provides a service for scanning websites for security vulnerabilities, regularly analyzes and reports on vulnerability data that it collects. Using data from 2013 and 2014, WhiteHat found that 35 percent of finance and insurance websites are “always vulnerable,” meaning that these sites had at least one serious vulnerability exposed every single day of the year. The stats for other industries and government organizations were even worse. Only 25 percent of finance and insurance sites were vulnerable for less than 30 days of the year. On average, serious vulnerabilities stayed open for 739 days, and only 27 percent of serious vulnerabilities were fixed at all, because of the costs and risks and overhead involved in getting patches out.

Continuous Delivery, and collaboration between developers and operations and infosec staff working closely together, can close vulnerability windows. Most security patches are small and don’t take long to code. A repeatable, automated Continuous Delivery pipeline means that you can figure out and fix a security bug or download a patch from a vendor, test to make sure that it won’t introduce a regression, and get it out quickly, with minimal cost and risk. This is in direct contrast to “hot fixes” done under pressure that have led to failures in the past.

Speed also lets you make meaningful risk and cost trade-off decisions. Recognizing that a vulnerability might be difficult to exploit, you can decide to accept the risk temporarily, knowing that you don’t need to wait for several weeks or months until the next release, and that the team can respond quickly with a fix if it needs to.

Speed of delivery now becomes a security advantage instead of a source of risk.

The Honeymoon Effect

There appears to be another security advantage to moving fast in DevOps. Recent research shows that smaller, more frequent changes can make systems safer from attackers by means of the “Honeymoon Effect”: older software that is more vulnerable is easier to attack than software that has recently been changed.

Attacks take time. It takes time to identify vulnerabilities, time to understand them, and time to craft and execute an exploit. This is why many attacks are made against legacy code with known vulnerabilities. In an environment where code and configuration changes are rolled out quickly and changed often, it is more difficult for attackers to follow what is going on, to identify a weakness, and to understand how to exploit it. The system becomes a moving target. By the time attackers are ready to make their move, the code or configuration might have already been changed and the vulnerability might have been moved or closed.

To some extent relying on change to confuse attackers is “security through obscurity,” which is generally a weak defensive position. But constant change should offer an edge to fast-moving organizations, and a chance to hide defensive actions from attackers who have gained a foothold in your system, as Sam Guckenheimer at Microsoft explains:

If you’re one of the bad guys, what do you want? You want a static network with lots of snowflakes and lots of places to hide that aren’t touched. And if someone detects you, you want to be able to spot the defensive action so that you can take countermeasures.... With DevOps, you have a very fast, automated release pipeline, you’re constantly redeploying. If you are deploying somewhere on your net, it doesn’t look like a special action taken against the attackers.

¹ Rich Smith, Director of Security Engineering, Etsy. “Crafting an Effective Security Organization.” QCon 2015 http://www.infoq.com/presentations/security-etsy

² “Put your Robots to Work: Security Automation at Twitter.” OWASP AppSec USA 2012, https://vimeo.com/54250716

³ http://www.slideshare.net/jallspaw/go-or-nogo-operability-and-contingency-planning-at-etsycom