The Rise of Continuous Packaging

Chapter 4. Continuous Packaging

Continuous packaging is a checkpoint that all packages, images, and artifacts used in and produced from your build processes and deployments pass through so that you can manage them consistently. Continuous packaging provides security, control, and visibility over all incoming and built assets.

Continuous packaging brings consistency and order to the steps between CI and CD, minimizing or eliminating ad hoc processes, bespoke scripts, unlogged manual procedures, and the use of libraries and packages without knowing their origins.

Continuous Packaging and CI/CD

The modern software development workflow is a DevOps pipeline that uses CI/CD to manage software delivery at scale. With continuous packaging, the CI/CD pipeline falls into three major phases:

Continuous integration

Developers write code that is continuously scanned and integrated before being packaged and pushed to a universal package repository.

Continuous packaging

All packages produced or consumed by the software supply chain are scanned, verified, and isolated from untrusted sources.

Continuous delivery

Packages are promoted through a series of tests, then delivered to customers where they can be deployed on customer or cloud infrastructure.

CI tools take source code changes as they come in, resolving dependencies on other packages or libraries, and build artifacts to be consumed by the CD tools. CI enforces consistency and automates integration, helping reduce problems such as code that works only on one particular developer’s computer.

Continuous packaging treats these assets as first-class traceable entities in the supply chain’s bill of materials, making it possible to track their origins and any changes along the way. All packages, images, and other artifacts used or produced by the build process pass through this checkpoint.

This means that continuous packaging tools must universally store and deliver packages in many different formats, making them available to development, staging, and production environments. During testing, the goal is to make sure each package is the same across all environments to ensure that they are tested consistently and comprehensively. Figure 4-1 shows the phases of CI/CD with continuous packaging.

Figure 4-1. Phases of CI/CD with continuous packaging

Continuous packaging helps with the challenges of dependency resolution by reducing the number of potentially incompatible or conflicting packages. As artifacts come in, their signatures are verified and they are scanned for vulnerabilities and malware. These processes reject bad packages, ensuring that the packages made available in the repository are trustworthy. When software is ready for release, continuous packaging processes make it available in a public or private universal repository that contains packages and their metadata, information that helps identify and verify them.

Providing access to the gamut of packages through one centralized store makes it simpler for clients to access them and helps normalize access and performance. Everyone who accesses the store gets the same experience, regardless of package type. Keeping all the different package types in a single repository also simplifies integration with build processes and reduces the number of locations your team needs to access reliably.

How Continuous Packaging Works

Continuous packaging establishes a trusted single source of truth for assets of differing types by providing three important factors:

• Provenance

• Isolation

• Universality

The following sections provide detail about how continuous packaging approaches each of these areas.

Provenance

Provenance is the ability to prove that packages, images, containers, and other artifacts are what they say they are, that they originate from the correct source, and that they have not been falsified or adulterated along the way. Provenance brings the auditability to packaging that version control brings to cryptographically verifiable source code commits.

The goal of provenance is an unbroken chain of traceability, all the way back to the host machine that built the package. Package metadata provides information such as the version, code contributors, scan results, license information, and build environment. This information is auditable, making it possible to integrate with other security tools.

Automated continuous packaging processes use checksums, cryptographic signatures, and other metadata to verify the identity and integrity of assets and establish that every package is genuine, signed, and matches the original. This helps keep software more secure, reducing vulnerabilities and exploits that could affect the code during the build process.

Not only does the chain of metadata help guarantee that packages are genuine, it can also provide information to developers, managers, and other stakeholders about how and why a feature or component came to be where it is. If a library or package version is involved in an unusually large number of problems, the provenance of the code in question can reveal how old it is, who has been working on it recently, and which features or fixes it implements.

Isolation

Much of the software built in a CI/CD pipeline depends on external packages, which can pose a security risk. Continuous packaging reduces attack vectors by isolating the software development pipeline from upstream package repositories, maintaining copies of the external packages in the private universal repository so that developers don’t access the external packages directly.

By caching packages from public upstream repositories and other external sources, you can protect yourself from issues with package availability, license changes, and emerging vulnerabilities. Package isolation provides a quarantine area where you can ensure that your code only accesses clean, known assets. One isolation approach is to maintain two private repositories: an unsafe repository that caches packages from a public upstream repository, and a safe repository with upstream proxying and caching disabled.

When the unsafe repository pulls packages from the upstream repository, continuous packaging automation uses a private repository management API to obtain package checksums and signatures. If the packages are genuine and pass vulnerability scans, they can be promoted to the safe repository, ready for use by development teams and for integration into the software build and deployment processes.

Internal build processes and development teams never connect directly to a public upstream repository. This closes off an important attack vector and provides the additional benefit of version and license control on all packages used by the software development life cycle. When an upstream repository releases a package with an undesirable change, you can freeze the version in the internal safe repository to prevent impact on the code in development.

Universality

To secure a DevOps pipeline, you must be able to manage packages together consistently, regardless of their formats and languages. This ensures transparency, visibility, and control across the entire inventory of assets. Continuous packaging provides a universal private repository that handles the protocols for different types of packages natively. The universal repository acts as a single source of truth, making assets available to different teams and for different deployments consistently from one place.

By ensuring that all packages pass through this single checkpoint, continuous packaging controls both the ingress and egress of the entire inventory, letting you see exactly what you’re consuming and what you’re building.

Benefits and Costs of Continuous Packaging

At the core of continuous packaging is package management, which is already present in nearly every software development pipeline. This means that continuous packaging tools for control, visibility, analytics, and accelerated delivery rest on package management foundations you already have in place.

Benefits of Continuous Packaging

Continuous packaging adds security, control, visibility, and performance to the management of software packages, whether they are external dependencies, build artifacts, or deployable releases. By managing packages of all types in a single repository, continuous packaging gives you visibility into all assets, helping reduce redundancy and discrepancies between package versions.

Isolating the repository from its upstream sources helps prevent vulnerabilities and attacks by providing an opportunity to test and scan incoming packages before making them available to developers and to the software build pipeline. Automatic scanning, checksums, and signatures ensure that the packages are genuine and unadulterated before you promote them through your development and testing environments or deploy them to production.

You can limit access to specific authorized clients and users inside or outside the company, without sharing packages to the world. Providing a single source of truth also reduces the complexity of the interaction between the CI and CD parts of the software supply chain. Maintaining fewer versions simplifies dependency resolution by constraining choice, thereby helping alleviate problems such as package conflicts or mutually incompatible versions.

The ability to keep specific versions of all packages in a single universal repository helps teams collaborate more easily and with less confusion. At the same time, because the repository handles the package management protocol for each asset type, it still works as expected with your existing package management tooling. This means you can combine packages of different types in any way that makes sense for the team.

You can use the same repository to distribute software to customers in different formats as well, which makes the publishing process simpler.

Caching packages at the edge gives development teams all over the world an optimal experience with low latency, without giving up control. Today, teams of developers distributed around the world must work together remotely. Every person needs to be able to communicate, collaborate, and work in the same environment in real time. Package management protocols exchange high volumes of metadata and other information, requiring the minimization of latency and network transmission time where possible. Requiring centralized access via a corporate LAN or a specific cloud region would create a bottleneck for globally dispersed teams through increased latency or high network load on specific servers.

Implementing continuous packaging as a method of managing packages globally can help mitigate these challenges by providing every team member with the same experience while distributing the load geographically. A single central repository with vetted, scanned packages becomes the center of a package delivery network that caches copies of all artifacts at edge servers located near each team.

This delivery network is similar to a standard content delivery network, but built specifically for software assets. The central repository and all edge caches are context aware and understand the package management protocols necessary for all the different package types. The edge handles authentication for private packages directly, further reducing the burden on the central repository. Bringing the packages closer to distributed teams helps ensure low latency for everyone as they collaborate on large software projects, no matter where in the world they’re located.

Costs of Continuous Packaging

The additional costs of continuous packaging are similar to the tooling costs for CI.

Mainly, you’ll need a place to host a universal package repository, preferably as a service so that you don’t have to take on management tasks. If you aren’t already using CI/CD as the foundation of your software development process, you’ll need to build some additional infrastructure to support a DevOps pipeline. You’ll need a CI server that can monitor your code in source control, running automated tests on every new commit. The team will need to write automated tests for every feature or bug fix. Developers will need to get in the habit of merging their changes at least once a day, more often if possible.

By managing the complexity between the CI and CD parts of the DevOps development pipeline, continuous packaging helps provide security to the software supply chain through source and build integrity.

Conclusion

As software development becomes increasingly distributed, with geographically dispersed teams building independent components, application architecture will rely on increasingly complex supply chains. Because this complexity introduces opportunities for risk, sophisticated techniques will be required to ensure that supply chains remain secure and reliable.

Software supply chain security is crucial for companies that want to protect themselves from financial loss, liability, and the exfiltration of sensitive data. It’s also a matter of national and international importance. To mitigate threats and risks to the software supply chain, maintaining the integrity of source code, packages, and dependencies is key. Nearly every point on the software supply chain is susceptible to attack, from dependencies through deployment. Because even the best-known third parties can be vulnerable to attack, verifying source code and packages, rather than trusting them, is necessary at every step of the way.

Cloud native architecture is the platform that supports the horizontally scalable applications that have become an indispensable part of the world economy. Microservices as a development model make the CI/CD software pipeline practical, decoupling the work of different teams so that application development is faster and more robust. Service meshes help manage growing numbers of independent microservices, relying on orchestrated containers for elastic, self-healing infrastructure. RESTful APIs throughout the pipeline make it possible to automate important security, packaging, testing, and development tasks consistently. But the cloud native software supply chain is not without its challenges.

Like any other supply chain, software development pipelines can be vulnerable to flaws, mistakes, or threats. Often, the majority of an application comprises open source or proprietary libraries or packages that are themselves the output from another supply chain. These interdependencies open up possibilities for numerous effective attack vectors. Between the CI and CD steps, there are many assets in a many-to-many relationship with each other. There have traditionally been few tools for managing packages at this point in the supply chain. To build secure software development pipelines with a DevOps approach, you must be able to manage, store, and deliver packages securely and efficiently across all package formats and languages.

Continuous packaging provides security, control, and visibility over all incoming and built assets in the form of a checkpoint that all packages, images, and artifacts used in and produced from your build processes and deployments pass through. In this way, continuous packaging brings consistency and order to the steps between CI and CD, minimizing or eliminating ad hoc processes, one-off scripts, unlogged manual procedures, and the use of libraries and packages without knowing their origins. By ensuring that all packages pass through this single checkpoint, continuous packaging controls both the ingress and egress of the entire inventory, letting you see exactly what you’re consuming and what you’re building.

Continuous packaging adds security, control, visibility, and performance to the management of software packages, whether they are external dependencies, build artifacts, or deployable releases. By managing packages of all types in a single repository, continuous packaging gives you visibility into all assets, helping reduce redundancy and discrepancies between package versions. Most importantly, continuous packaging provides a single source of truth for packages, containers, and other assets consumed by the development process, isolating the supply chain from upstream sources and giving organizations control.

As software architecture continues to evolve toward faster delivery, further complexity will ensue. Most applications already rely on tens or hundreds of libraries, each of which introduces possible vulnerabilities and attack surfaces. Continuous packaging helps bring new ways to ensure the reliability and security of the software supply chain through provenance, isolation, and universal package management, helping accelerate application development safely.