Implementing continuous delivery
The architectural design, automated quality assurance, and deployment skills needed for delivering continuous software.
The architectural design, automated quality assurance, and deployment skills needed for delivering continuous software.
There is an ever-increasing range of best practices emerging around microservices, DevOps, and the cloud, with some offering seemingly contradictory guidelines. There is one thing that developers can agree on: continuous delivery adds enormous value to the software delivery lifecycle through fast feedback and the automation of both quality assurance and deployment processes. However, the challenges for modern software developers are many, and attempting to introduce a methodology like continuous delivery—which touches all aspect of software design and delivery—means several new skills typically outside of a developer’s comfort zone must be mastered.
These are the key developer skills I believe are needed to harness the benefits of continuous delivery:
Let’s explore each of these skills and I’ll also offer some resources for mastering them.
I began my career creating monolithic Java EE applications. I took requirements specified in an issue tracker and coded both the frontend and backend implementation of the requirements. However, testing and quality assurance was conducted by a separate team. If a new feature was successfully validated, then yet another independent ops team packaged the application and deployed the artefact into an application server that was running on pre-built co-located infrastructure.
Things couldn’t be more different now, and I learned this the hard way as I began working for a series of startups in the mid 2000s. Due to limited resources, the CTO or CEO of these small companies would often ask if I could also deploy and configure hardware, VMs, and databases. I was happy to learn more, and becoming a part-time sysadmin was a great challenge. Learning these new skills while also creating applications that satisfied emerging business requirements was even more challenging. Fast forward ten years, and now developers are increasingly expected to work closely with the business team, perform their own QA, and deploy to a platform via a self-service user interface. Some teams are empowering developers to also be responsible for building and operating the platform infrastructure. This is personified in the DevOps philosophy: we must all share responsibility for the delivery of value to our stakeholders.
The shared responsibility that comes with the continuous delivery of effective software means developers need strong architectural design skills, both in terms of providing an effective structure to satisfy current requirements and afford options for evolution, and also to satisfy the operational requirements of the platform the application is being deployed to. So, what does that mean in concrete terms and how can you begin to master these skills?
Architecture if often referred to “as the stuff that is difficult or expensive to change,” and at the core of this are the concepts of cohesion and coupling. We can see evidence of this in Adrian Cockcroft’s definition of microservices as “loosely coupled service-oriented architecture (SOA) with bounded contexts.” The concept of bounded contexts, taken from Domain-Driven Design (DDD), is fundamentally about modelling and building systems as highly cohesive contexts, or services, which in turn facilitates continuous testing as we can focus on validating logical (and physically) grouped functionality.
Loose coupling enables services to be continuously deployed independently and in isolation, which is critical for maintaining a high velocity of change for the system as a whole.
In addition to learning about the core principles of high cohesion and loose coupling, you will also benefit from mapping this knowledge to API design and creating “cloud native” applications. Good API design is difficult, but exposing application functionality through a cohesive well-defined interface promotes easier testing throughout the build and QA processes, and encourages developers to design services “outside-in”; starting with the business requirements.
Embracing cloud-native architectural practices, as typified by Heroku’s twelve-factor application, also promotes continuous delivery by separating deployment and operational configuration from application code, and thus loosely coupling an application with its environment.
Continuous testing is an essential tool in your development skill set. Why should you test software? The answer that first jumps to mind is to ensure that you are delivering the functionality required, but the complete answer is actually more complex. You obviously need to ensure that software is functionality capable of doing what was intended—i.e. delivering business value—but you also need to test for the presence of bugs, to ensure that the system is reliable and scalable, and in some cases cost-effective. This validation cannot be a one-off process because software is inherently mutable and coupled, and the smallest change can often cause a cascade of effects. This is especially true of applications that use the microservices architectural style, as this type of application is inherently a distributed and complex adaptive system. Continuous testing is therefore an essential tool in your development skill set.
It is important that we are clear of the types of testing we must perform on our system, and how much of this can be automated within a continuous delivery build pipeline. Classically, validating the quality of a software system has been divided into two types: functional and nonfunctional requirements. Be warned that although the traditional phrasing of “nonfunctional requirements” may make these quality attributes appear less important. They are not. I have seen many systems that were not functional due to nonfunctional requirements!
A useful introduction to understanding the types and goals of testing can be found in “Agile Testing: A Practical Guide for Testers and Agile Teams” (Addison Wesley) by Lisa Crispin and Janet Gregory. The entire book is well-worth reading, but the most important concept for this article is the Agile Testing Quadrants, based upon original work by Brian Marick. This a 2×2 box diagram shown below, with the x-axis representing the test purpose—from guiding development to critiquing the product—and the y-axis representing what the test targets—from technology facing to business facing.
Let’s take a look at how the Agile Testing Quadrants apply to continuous delivery and some of associated tools and techniques.
Quadrants Q1 and Q2 (clockwise from the lower left) are focused on guiding development from both technological and a business perspectives, and these tests are highly automatable. Approaching the development of functionality from outside-in allows tooling such as Serenity BDD to be used to define story (acceptance) tests which make use of business-friendly DSLs such as Cucumber or JBehave. Here are a few examples:
Quadrant Q4 in the diagram is technolog facing and focused on critiquing the product—this is the place for NFR testing. Core tooling and skills to master here includes:
The vast majority of all the tooling mentioned within this section is API or CLI-driven, and can therefore with a reasonable amount of effort be incorporated into a build pipeline for continual validation. Due to space limitations in this article, we won’t cover the details of quadrant Q3. Although the techniques mentioned in Q3 are vital for the successful delivery of business value, they are inherently focused around manual processes such as exploratory testing that are not implemented within a continuous delivery build pipeline. Additional information on Q3, and all of the quadrants, can be found in the Agile Testing Essentials LiveLessons on Safari, which is authored by Lisa Crispin and Janet Gregory.
Now with both architecture and automated testing skills in outlined, let’s focus on the delivery of valuable software to various environments, including production.
The requirements for increased velocity and the associated modern architectural styles strongly encourage separating the process of deployment and release. This has ramifications for the way you design, test and continuously deliver software.
One core approach to such deployment is the use of feature flags or ability to turn features (sub-sections) of your application on/off with ease. This allows you to continuously deploy new features, but control when they are released into production. Sam Newman, author of Building Microservices (O’Reilly), has created a talk that explains the principles and practices of feature flags in great detail.
Understanding the platform on which your applications will be deployed, allows “mechanical sympathy”—the ability to understand and extract the best performance from the underlying infrastructure—and shows what deployment approaches are available to you.
Modern platforms typically allow a range of deployment types:
Each type of deployment can impact core architectural and persistence decisions in its own way: a highly coupled application may only allow all-at-once or blue/green deploys even with the use of feature flags to control the actual releases; or an upgrade that uses rolling deployment but changes an underlying database or API schema may need to use the multiple-phase upgrade pattern.
A developer’s responsibility also doesn’t end with the deployment and release of functionality: appropriate logging, metrics and alerting must be incorporated into the codebase. This not only assists the developer during building and testing, but critically also assist with monitoring and diagnosing of production performance and issues. Modern languages and platforms provide Metrics libraries often utilizing or ported from Coda Hale’s classic Java Metrics library, such as Spring Boot’s Actuator, Metrics.NET and go-metrics. This data can be collected and analyses by cloud native tooling such as fluentd and Prometheus, the ELK stack, or commercial tooling like Humio.
The majority of cloud providers now also provide distributed tracing systems that can help developers understand the flow of a request through the entire system, for example AWS X-Ray and AWS CloudWatch; Google Stackdriver Trace and Stackdriver Logging. Live debugging of incidents (ideally within non-production environments) can be assisted by using tooling that gives further operational insight at runtime, such as Datawire’s Telepresence, Sysdig, and docker/kubectl exec.
Experience from debugging production incidents with the above tooling has taught me that there is nearly always a lesson to be learned or a test to be created that can be retrofitted into the build pipeline to prevent a future regression.
When I began my career in software development I didn’t truly know what essential skills were needed to become professional in this domain. After fifteen years of experience I’ve learned that to be an effective developer you must be competent with skills in architecture, testing, deployment and the operation of software. No one is saying that this is easy, but your ability to create valuable software is directly impacted by continual improvement of both fundamental skills and techniques relating to the changing technical landscape. I’m sure in the next fifteen years of my career I’ll realize that even more skills are required.
Continuous delivery principles and practices are a fantastic approach to increase feedback for both developers and the business stakeholders, and also to continually assert functionality and system properties as the software changes and evolves. These principles are closely related to the DevOps philosophy, which is fundamentally about increasing shared responsibility and end-to-end accountability, with the ultimate goal of enabling the continual delivery of valuable software to users.