Cloud Native Application Security

Chapter 4. Adapting to Dev-First CNAS

Adopting CNAS requires significant changes to the way we secure applications and infrastructure. Making the shift is a journey, one that is different for every organization, and even for different parts of the same org.

Although choosing the right path is your decision to make, patterns and best practices are starting to emerge for getting it right. In this chapter, I suggest a few areas in which to consider breaking the status quo, and how.

Rethinking the Security Org Structure

Organizations are typically split based on areas of responsibility. As you shift your view to see securing parts of your infrastructure as an application security problem, reconsider how you structure the security org. More specifically, consider whether to change the scope of responsibility for the application security team.

In addition, as your security practices become more dev-first and focus on empowerment of developers, your requirements for this AppSec team change. You need more empathy and program management alongside more engineering capacity. You need more builders and fewer breakers.

To help you evaluate your security organization structure, here are the three most common team scopes I see in the AppSec world: Core AppSec, Security Engineering, and the newer Product Security scope. These should serve as reference points for how to structure your org, not perfect models to adopt.

Core Application Security Team

Let’s start from the status quo, keeping the same scope for the application security team. Since this is the default state, most organizations use this team scope—at least as a starting point.

The mandate of the core AppSec team is securing the custom application code and business logic as well as the open source libraries being used. They typically own the classic Application Security Testing (AST) suite, including static, dynamic, and interactive AppSec testing (SAST, DAST, and IAST) to find vulnerabilities in custom code, and software composition analysis (SCA) tools to find vulnerable open source libraries. In addition, these teams often develop security education and training and, potentially, vulnerability management or bug bounty. In some cases, they may own runtime application protection too, using RASP or WAF tools.

Core AppSec team members typically need to be subject matter experts in secure coding and have some experience running audits and security code reviews. They need good developer empathy to collaborate with dev, which in turn requires some ability to understand or relate to code but doesn’t require full software development credentials.

The main advantage of sticking to a core AppSec team is the tenure it has in the industry. For hiring, it makes it easier to hire professionals who bring with them experience across the entire team domain. For tooling, it’s a space where the tools and practices are well documented. And from an organizational perspective, most of the industry will assume an AppSec team looks like this core AppSec team.

Although the scope of a core AppSec team is the status quo, its methodologies have often become more dev-empowering. AppSec teams often assign individuals on the team to be the partner to several dev teams, helping foster better collaboration. Many others run Security Champions programs, helping them get scale and embed more security expertise in the dev team. Although the scope is mostly unchanged, the internal practices of core AppSec teams don’t have to be traditional.

Security Engineering/Platform Team

Automating security steps is key in a modern development environment. Fast CI/CD pipelines leave no room for manual review, requiring automated pipeline tests instead. In addition, developers are not security experts; they have less time to devote to security and, thus, need tools that have embedded security expertise and can offload or facilitate security decisions.

Building and operating security tools is no easy feat, especially in large organizations where different dev teams have vastly different requirements. To help boost automation, some organizations created dedicated security engineering teams that focus on building internal tools and integrating external tools, all meant to bolster security.

Security engineering teams are made up of software engineers with a slight bias for security, and operate like a complete DevOps engineering team. They typically build, deploy, and operate the services they build, and use the same methodologies other engineering teams use to run their agile processes and manage product backlog.

If the volume of work isn’t big enough to warrant its own team, this same activity is typically embedded in the core AppSec team. However, though teams titled Security Engineering are pretty consistent in their charter, individuals with the (increasingly common) Security Engineer title vary greatly in their responsibilities. Some are software engineers as described, whereas for others the “Engineer” part of the title refers to the security realm instead.

Security engineering teams are a great way to truly level up the amount of automation and are a great parallel team to ops-oriented platform or site reliability engineer (SRE) teams. In fact, in quite a few cases, the platform team’s scope has been expanded to include building and operating such security tools. It’s also a great way to get software engineers into the security team, helping with the talent shortage problem and with building more dev empathy in the security team.

Product Security/Cloud Native AppSec Team

A more recent addition to the security team patterns is the product security team. These teams have a bigger scope that includes not only the application code itself, but everything to do with the product. Most notably, two key additions are capturing the full CNAS scope and helping build security features in the product itself.

Full CNAS scope

Growing to include the CNAS scope is a natural result of rethinking certain infrastructure risks as application security. As discussed in Chapter 3, technologies like containers and IaC are driven by the same developers writing custom code and using the same practices and tools. To support this change, AppSec teams need to support those engineers in doing so successfully. Teams that embrace this broader scope often refer to themselves as product security teams.

This expanded CNAS scope means product security teams work across a bigger portion of the software development life cycle. It includes more engagement with production deployments and even operations, leading to overlap with the more ops-focused cloud security team. In practice, cloud native development means that cloud security is affected by both dev and ops teams, and product security teams cover the former.

Note that many core AppSec teams are expanding to embrace the full CNAS scope without officially changing their name and mandate. Choosing and implementing solutions to scan container images for vulnerabilities and audit IaC files is increasingly the AppSec team’s domain. Although it’s safe to assume product security teams capture this complete scope, such a rename is not strictly necessary, and many AppSec teams have evolved without such a declaration.

Product security features

A bit unrelated to CNAS, but still noteworthy, is the involvement of product security teams with a more user-oriented part of security: security features. As user awareness of the importance of security grows, many products look to build dedicated security features and differentiate through them. Deciding what security capabilities are of value requires a level of security understanding that dev teams may not have, but security teams do. Product security teams often embrace an explicit role here, collaborating with product managers (PMs), who own the complete product functionality and value proposition, more than ever before.

This responsibility plays an important role in the relationship between application and security teams. Security controls are a means to mitigate risk, but being able to present this risk mitigation as a security feature means it can help grow revenue instead. Growing revenue is another shared goal for the two teams, and one that is far more visible than risk reduction, making it easier to celebrate success.

The evolution of product security

Product security is a new title and scope and is still being defined. Given its broader scope, it’s often a parent title or group, which encompasses the other mentioned teams. In some cloud native organizations, product security is the primary scope of the chief security officers (CSOs), whereas a few others started naming leaders as chief product security officers (CPSOs).

Adrian Ludwig, Atlassian’s chief information security officer (CISO), phrased it best, saying, “Product security’s goal is to improve the security posture of products and represent customers’ security expectations internally to the development team”. The title is used by other companies such as Twilio, Deliveroo, and Snyk, and I believe it’s the right way to address CNAS.

What About the DevSecOps Team?

You may have noticed I didn’t name a DevSecOps team, and that wasn’t by chance. Like DevOps, DevSecOps isn’t a team; it’s a movement, meant to embed security into the core dev and operations work. In my opinion, it shouldn’t be the title of a team.

However, just like DevOps teams exist, so do DevSecOps teams—and their mandates vary greatly. At times, they’re really a cloud security team, focusing on operations and runtime security. Other times, they’re more platform-like, with scopes similar to security engineering teams. Since the title doesn’t imply a specific set of responsibilities, the scope of a DevSecOps team isn’t something that can really be defined.

However, what all those teams have in common is that they are forward thinking. DevSecOps aims to change how we do security, and DevSecOps teams, wherever their scope, consistently see themselves as change agents. They embrace automation and cloud, favor engineering security solutions over running audits, and aim to empower dev and ops teams to secure their work on their own.

Rethinking Tooling

Alongside organizational structure changes, CNAS and dev-first require a reevaluation of your toolkit. Given this new perspective, you should reconsider the most important traits you’re looking for in a technology solution and how you want tools to be bundled.

There are many ways to reevaluate tools, but I would suggest focusing on three primary areas: dev adoption, platform scope, and governance approach.

Developer Tooling Caliber

If our goal is to get developers to build security into their daily work, we need to make sure we equip them with tools that optimize for that goal. If you buy solutions designed for auditors and ask developers to use them, you’re unlikely to be successful.

Developers, unsurprisingly, are used to using developer tools. These tools represent an entire industry, which evolved its own best practices around what makes a great developer tool. To help you choose a security solution developers will embrace, you should evaluate these tools against developer tooling best practices and see how they fare.

Here are a few of the common traits of a great developer tool:

Successful self-serve adoption

Practically all successful dev tools have great self-serve usage, including easy onboarding, intuitive workflows, and great documentation. This is the way developers like to consume tools, and it forces vendors to ensure that their tools relate to developers without a salesperson pushing it. Unless you want to be that salesperson promoting a tool to your developers, look for tools with a proven track record of self-serve adoption by dev.

Seamless integration into workflows

Dev tools, for the most part, look to meet developers where they are instead of asking them to open yet another tool. They integrate elegantly into their IDE, Git, and pipelines and provide value at the right point in time. Integrations are about more than technical hooks; they need to adapt to workflows and best practices, or they will be rejected.

Rich API and automation friendly

Although opinionated integrations are needed to get started, a dev tool must also be a Swiss Army knife. A rich API and automation-friendly client (CLI/software development kit [SDK]) are mandatory, both to tune the tool to every pipeline and to allow the community to build on it. If you can’t build on a tool, it’s not a great dev tool.

Adoption by open source and community

Developers look to fellow developers to validate a tool’s credibility, and open source adoption is the best indicator of that. Seeing open source projects integrate a security tool or open source projects that build on it are great developer community validations. When inspecting a security tool, inspect its adoption amid open source and draw your own conclusions.

These are just a few of many dev tool best practices. They should be added to the security-specific recommendations around dev-first security from Chapter 2. In addition, make sure you involve actual app developers when evaluating any tool, to get a real-life perspective on how well it fits into their surroundings (see Figure 4-1).

Figure 4-1. Examples of developer-friendly security tools

Platform Scope

AST platforms today are primarily focused on custom code and, perhaps, the open source libraries it consumes. Tool suites are primarily composed of SAST, DAST, and IAST, with SCA recently added to the mix. As you embrace the broader cloud native application scope, reconsider the makeup of the platform.

To start, let’s consider what tools benefit from being part of a single platform. They can be divided into a few big buckets:

Shared users

If different tools are meant for different primary users, there’s little need for them to be part of a single platform because they’ll be used separately anyway.

Shared workflows

If multiple tools are used in a similar fashion and integrate in similar points of the user’s workflow, you can save the integration time and user time and effort by combining them instead of requiring engagement with multiple separate tools.

Shared prioritization

If the action items from different tools should be prioritized against one another, sharing a backlog can increase efficiency and results.

Multiplied value

Finally, the strongest driver for tools to share a platform is when using tools together can enhance each tool.

This criterion naturally explains why technologies like SAST and SCA fit well in a single platform. Both serve the same developer users, run scans and engage users in the same spots, and share a backlog of security vulnerabilities the same developer will need to prioritize between. In advanced SCA solutions, SAST technology can indicate whether your custom code invokes the vulnerable code in a library, providing greater accuracy—hence multiplied value.

The same logic applies when you consider adding container and IaC security to the same platform of SAST and SCA:

Shared users

Containers and IaC are now secured by developers, just like SAST and SCA, who would rather not have multiple disparate products to learn and engage with.

Shared workflows

Securing containers and IaC requires integration across the life cycle, such as IDE, Git, and build integrations, just like SAST and SCA.

Shared prioritization

The same developer needs to spend cycles fixing a container or IaC vulnerability or one in the code or libraries—all one backlog for securing the app.

Multiplied value

There are many ways in which securing these components combine. Scanning containers requires exploring the app inside, knowing the infra config helps prioritize code and library vulnerabilities, and understanding the flow across components helps mitigate issues; this is precisely what you do when triaging a vulnerability.

This logic continues to hold even as the CNAS scope expands, and I encourage you to think about your CNAS tooling as one platform. An easy guideline is that anything sitting in a Git repository probably relates to the files around it and follows the same development workflows. As a result, a CNAS platform should help secure everything in the repo—the entire cloud native app.

DAST and IAST are slightly awkward participants in such a platform. Although they deal with securing the app, they typically require different workflows; it’s hard to fit them into a build pipeline or IDE scans due to the time they take to run. In my opinion, this is a technical problem, not a logical one, which will hopefully be solved once IAST and DAST solutions evolve to become more pipeline friendly.

Governance and Empowerment Approach

Embracing a dev-first security approach requires a different type of collaboration. Instead of having tools that focus on broadcasting and enforcing the top-down mandate, we need tools that help us build secure applications collaboratively.

This may sound obvious, but in practice it’s a big departure from the way many security tools work. Let’s dig into a few concrete examples to get a taste of what this means.

First, developers need to be empowered to balance business needs with security risk. This means, for example, allowing them to decide not to fix a discovered vulnerability and continue deploying it to production. This is a scary proposition to some, but it’s the only way to enable independent teams. Note that ignoring a vulnerability should still require a provided (and auditable) reason, and certain constraints should be hard lines (typically for compliance reasons), but as a default stance, the decision should be left to the developer.

Second, developers need to be able to manage their security risk, and that requires seeing all the vulnerabilities in their projects. This needs to include not only the list of vulnerabilities, but also all the information needed to prioritize well, such as exploitability and asset mapping. Being transparent with such information may introduce some risk, but it’s the only way to scale security; to empower developers, you have to trust them with this sensitive data.

Third, security teams should invest in tracking and driving adoption of security controls, even more than their outputs. Dev teams should manage their vulnerability backlogs, but the security team needs to help them do so successfully. Dev-first security governance means tracking which controls are adopted and how well their output is handled—and working with dev teams to improve those stats as opposed to highlighting the vulnerabilities they should fix.

These are just a few examples of what to consider when assessing your governance tools and techniques. The key is to embrace a platform mentality; its goal is to help dev teams succeed in building secure software, not to track the security flaws themselves.

Rethinking Priorities

Last but not least, CNAS requires rethinking application security priorities. If developers need to secure the entire cloud native application, where do you want them to focus the most?

Historically, IT security controls dominated far bigger budgets and commanded more CISO attention than application security ones did. There may be historical reasons for this imbalance, but at the end of the day, it represents a CISO’s view on how best to use their dollars to reduce the risk to the organization.

In other words, CISOs believe the risk of a breach due to an unpatched server or misconfigured infrastructure is greater than that of a custom vulnerability in your code. This perception has a fair bit of data to support it, showing historical breaches and their causes. In addition, it’s easy to understand; it’s far easier for an attacker to run known exploits at scale and look for an open door than to reverse engineer an application and find custom flaws to let them through.

The cloud doesn’t diminish this equation; in fact, it strengthens it. Adoption of cloud means more infrastructure and more servers are used, they tend to be more publicly accessible, and they are defined by teams that are less familiar with managing them (developers) and equipped with less-mature tools.

However, whereas the previous balance was between IT security budgets and AppSec budgets, now it’s all in the AppSec world. When building cloud native applications, the same developers need to secure their custom code, manage vulnerabilities in their open source usage, and make the servers and infrastructure resilient. The same AppSec team needs to help them do so as well as split the same budget across them.

So we’re back to the opening question: how do you want your precious developer time and limited CNAS budget distributed?

Left to their own devices, application security budgets and developer attention focus on securing custom code. AppSec maturity models, industry best practices, and the personal experience of individuals on the team are all anchored in the pre-cloud world, where custom code was the majority of what developers had to secure. Buying tools like SAST and DAST is often the first thing on a new AppSec leader’s list and is rarely challenged.

However, given the CNAS scope, is that still the best use of your time and money? The risk of a breach due to a known vulnerability or a misconfiguration is still higher than that of a custom code flaw, even if developers are the ones configuring it. If you agree, shouldn’t you tell your developers and AppSec team to focus first and foremost on securing those layers, and only then approach their custom code?

This isn’t an easy question to answer. I won’t assume to know what your priorities should be, because those change from org to org. However, I strongly encourage you to have this conversation internally and reconsider your priorities, given the broader scope of CNAS.

Conclusion

Changing how you approach application security isn’t just a thought exercise. It has very real downstream implications, including people’s careers, budget allocations, and a reassessment of the best way to keep the organization secure. It requires shaking off some muscle memory of how you’ve done things in the past and instead going back to first principles when choosing solutions, which takes precious time and energy.

The good news is that you don’t have to do it all at once. The changes I outlined in this chapter are related to one another but can be applied at your own pace. I suggest you pick the ones you relate to the most, or opt for other changes you think are more important, and get the changes going. Step by step, you’ll adapt your security function to the cloud native reality.