Pipeline as Code

Pipeline as Code

by Mohamed Labouardy
Released October 2021
Publisher(s): Manning Publications
ISBN: 9781617297540

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Start thinking about your development pipeline as a mission-critical application. Discover techniques for implementing code-driven infrastructure and CI/CD workflows using Jenkins, Docker, Terraform, and cloud-native services.

In Pipeline as Code, you will master:

  • Building and deploying a Jenkins cluster from scratch
  • Writing pipeline as code for cloud-native applications
  • Automating the deployment of Dockerized and Serverless applications
  • Containerizing applications with Docker and Kubernetes
  • Deploying Jenkins on AWS, GCP and Azure
  • Managing, securing and monitoring a Jenkins cluster in production
  • Key principles for a successful DevOps culture
Pipeline as Code is a practical guide to automating your development pipeline in a cloud-native, service-driven world. You’ll use the latest infrastructure-as-code tools like Packer and Terraform to develop reliable CI/CD pipelines for numerous cloud-native applications. Follow this book's insightful best practices, and you’ll soon be delivering software that’s quicker to market, faster to deploy, and with less last-minute production bugs.

About the Technology
Treat your CI/CD pipeline like the real application it is. With the Pipeline as Code approach, you create a collection of scripts that replace the tedious web UI wrapped around most CI/CD systems. Code-driven pipelines are easy to use, modify, and maintain, and your entire CI pipeline becomes more efficient because you directly interact with core components like Jenkins, Terraform, and Docker.

About the Book
In Pipeline as Code you’ll learn to build reliable CI/CD pipelines for cloud-native applications. With Jenkins as the backbone, you’ll programmatically control all the pieces of your pipeline via modern APIs. Hands-on examples include building CI/CD workflows for distributed Kubernetes applications, and serverless functions. By the time you’re finished, you’ll be able to swap manual UI-based adjustments with a fully automated approach!

What's Inside
  • Build and deploy a Jenkins cluster on scale
  • Write pipeline as code for cloud-native applications
  • Automate the deployment of Dockerized and serverless applications
  • Deploy Jenkins on AWS, GCP, and Azure
  • Grasp key principles of a successful DevOps culture


About the Reader
For developers familiar with Jenkins and Docker. Examples in Go.

About the Author
Mohamed Labouardy is the CTO and co-founder of Crew.work, a Jenkins contributor, and a DevSecOps evangelist.

Quotes
Configuring CI/CD platforms has never been easier!
- Ubaldo Pescatore, PagoPA

A must-read for any aspiring and seasoned devops/release automation engineer.
- Giridharan Kesavan, Visa

A very useful resource, not only for setting up and using Jenkins for CI/CD, but also for understanding the importance of Packer, Terraform, Docker, and Kubernetes.
- Kosmas Chatzimichalis, Mach7x

A perfect journey through pipeline-based software delivery.
- Satej Kumar Sahu, Honeywell

A brilliant, hands-on deep dive into how to implement modern CI/CD pipelines.
- Matthias Busch, Otto GmbH

Table of contents

  1. inside front cover
  2. Pipeline as Code
  3. Copyright
  4. brief contents
  5. contents
  6. front matter
    1. preface
    2. acknowledgments
    3. about this book
      1. Who should read this book
      2. How this book is organized
      3. About the code
      4. liveBook discussion forum
      5. Other online resources
    4. about the author
    5. about the cover illustration
  7. Part 1. Getting started with Jenkins
  8. 1 What’s CI/CD?
    1. 1.1 Going cloud native
      1. 1.1.1 Monolithic
      2. 1.1.2 Microservices
      3. 1.1.3 Cloud native
      4. 1.1.4 Serverless
    2. 1.2 Defining continuous integration
    3. 1.3 Defining continuous deployment
    4. 1.4 Defining continuous delivery
    5. 1.5 Embracing CI/CD practices
    6. 1.6 Using essential CI/CD tools
      1. 1.6.1 Choosing a CI/CD tool
      2. 1.6.2 Introducing Jenkins
    7. Summary
  9. 2 Pipeline as code with Jenkins
    1. 2.1 Introducing the Jenkinsfile
      1. 2.1.1 Blue Ocean plugin
      2. 2.1.2 Scripted pipeline
      3. 2.1.3 Declarative pipeline
    2. 2.2 Understanding multibranch pipelines
    3. 2.3 Exploring the GitFlow branch model
    4. 2.4 Test-driven development with Jenkins
      1. 2.4.1 The Jenkins Replay button
      2. 2.4.2 Command-line pipeline linter
      3. 2.4.3 IDE integrations
    5. Summary
  10. Part 2. Operating a self-healing Jenkins cluster
  11. 3 Defining Jenkins architecture
    1. 3.1 Understanding master-worker architecture
    2. 3.2 Managing Jenkins workers
      1. 3.2.1 SSH
      2. 3.2.2 Command line
      3. 3.2.3 JNLP
      4. 3.2.4 Windows service
    3. 3.3 Architecting Jenkins for scale in AWS
      1. 3.3.1 Preparing the AWS environment
      2. 3.3.2 Configuring the AWS CLI
      3. 3.3.3 Creating and managing the IAM user
    4. Summary
  12. 4 Baking machine images with Packer
    1. 4.1 Immutable infrastructure
    2. 4.2 Introducing Packer
      1. 4.2.1 How does it work?
      2. 4.2.2 Installation and configuration
      3. 4.2.3 Baking a machine image
    3. 4.3 Baking the Jenkins master AMI
      1. 4.3.1 Configuring Jenkins upon startup
      2. 4.3.2 Discovering Jenkins plugins
    4. 4.4 Baking the Jenkins worker AMI
    5. Summary
  13. 5 Discovering Jenkins as code with Terraform
    1. 5.1 Introducing infrastructure as code
      1. 5.1.1 Terraform usage
    2. 5.2 Provisioning an AWS VPC
      1. 5.2.1 AWS VPC
      2. 5.2.2 VPC subnets
      3. 5.2.3 VPC route tables
      4. 5.2.4 VPC bastion host
    3. 5.3 Setting up a self-healing Jenkins master
    4. 5.4 Running Jenkins with native SSL/HTTPS
    5. 5.5 Dynamically autoscaling the Jenkins worker pool
      1. 5.5.1 Launch configuration
      2. 5.5.2 Auto Scaling group
      3. 5.5.3 Autoscaling scaling policies
      4. 5.5.4 Workers CPU utilization load
    6. Summary
  14. 6 Deploying HA Jenkins on multiple cloud providers
    1. 6.1 Google Cloud Platform
      1. 6.1.1 Building Jenkins VM images
      2. 6.1.2 Configuring a GCP network with Terraform
      3. 6.1.3 Deploying Jenkins on Google Compute Engine
      4. 6.1.4 Launching automanaged workers on GCP
    2. 6.2 Microsoft Azure
      1. 6.2.1 Building golden Jenkins VM images in Azure
      2. 6.2.2 Deploying a private virtual network
      3. 6.2.3 Deploying a Jenkins master virtual machine
      4. 6.2.4 Applying autoscaling to Jenkins workers
    3. 6.3 DigitalOcean
      1. 6.3.1 Creating Jenkins DigitalOcean Snapshots
      2. 6.3.2 Deploying a Jenkins master Droplet
      3. 6.3.3 Building Jenkins worker Droplets
    4. Summary
  15. Part 3. Hands-on CI/CD pipelines
  16. 7 Defining a pipeline as code for microservices
    1. 7.1 Introducing microservices-based applications
    2. 7.2 Defining multibranch pipeline jobs
    3. 7.3 Git and GitHub integration
    4. 7.4 Discovering Jenkins jobs’ XML configuration
    5. 7.5 Configuring SSH authentication with Jenkins
    6. 7.6 Triggering Jenkins builds with GitHub webhooks
    7. Summary
  17. 8 Running automated tests with Jenkins
    1. 8.1 Running unit tests inside Docker containers
    2. 8.2 Automating code linter integration with Jenkins
    3. 8.3 Generating code coverage reports
    4. 8.4 Injecting security in the CI pipeline
    5. 8.5 Running parallel tests with Jenkins
    6. 8.6 Improving quality with code analysis
    7. 8.7 Running mocked database tests
    8. 8.8 Generating HTML coverage reports
    9. 8.9 Automating UI testing with Headless Chrome
    10. 8.10 Integrating SonarQube Scanner with Jenkins
    11. Summary
  18. 9 Building Docker images within a CI pipeline
    1. 9.1 Building Docker images
      1. 9.1.1 Using the Docker DSL
      2. 9.1.2 Docker build arguments
    2. 9.2 Deploying a Docker private registry
      1. 9.2.1 Nexus Repository OSS
      2. 9.2.2 Amazon Elastic Container Registry
      3. 9.2.3 Azure Container Registry
      4. 9.2.4 Google Container Registry
    3. 9.3 Tagging Docker images the right way
    4. 9.4 Scanning Docker images for vulnerabilities
    5. 9.5 Writing a Jenkins declarative pipeline
    6. 9.6 Managing pull requests with Jenkins
    7. Summary
  19. 10 Cloud-native applications on Docker Swarm
    1. 10.1 Running a distributed Docker Swarm cluster
    2. 10.2 Defining a continuous deployment process
    3. 10.3 Integrating Jenkins with Slack notifications
    4. 10.4 Handling code promotion with Jenkins
    5. 10.5 Implementing the Jenkins delivery pipeline
    6. Summary
  20. 11 Dockerized microservices on K8s
    1. 11.1 Setting up a Kubernetes cluster
    2. 11.2 Automating continuous deployment flow with Jenkins
      1. 11.2.1 Migrating Docker Compose to K8s manifests with Kompose
    3. 11.3 Walking through continuous delivery steps
    4. 11.4 Packaging Kubernetes applications with Helm
    5. 11.5 Running post-deployment smoke tests
    6. 11.6 Discovering Jenkins X
    7. Summary
  21. 12 Lambda-based serverless functions
    1. 12.1 Deploying a Lambda-based application
    2. 12.2 Creating deployment packages
      1. 12.2.1 Mono-repo strategy
      2. 12.2.2 Multi-repo strategy
    3. 12.3 Updating Lambda function code
    4. 12.4 Hosting a static website on S3
    5. 12.5 Maintaining multiple Lambda environments
    6. 12.6 Configuring email notification in Jenkins
    7. Summary
  22. Part 4. Managing, scaling, and monitoring Jenkins
  23. 13 Collecting continuous delivery metrics
    1. 13.1 Monitoring Jenkins cluster health
    2. 13.2 Centralized logging for Jenkins logs with ELK
      1. 13.2.1 Streaming logs with Filebeat
      2. 13.2.2 Streaming logs with the Logstash plugin
    3. 13.3 Creating alerts based on metrics
    4. Summary
  24. 14 Jenkins administration and best practices
    1. 14.1 Exploring Jenkins security and RBAC authorization
      1. 14.1.1 Matrix authorization strategy
      2. 14.1.2 Role-based authorization strategy
    2. 14.2 Configuring GitHub OAuth for Jenkins
    3. 14.3 Keeping track of Jenkins users’ actions
    4. 14.4 Extending Jenkins with shared libraries
    5. 14.5 Backing up and restoring Jenkins
    6. 14.6 Setting up cron jobs with Jenkins
    7. 14.7 Running Jenkins locally as a Docker container
    8. Summary
    9. Wrapping up
  25. index

Product information

  • Title: Pipeline as Code
  • Author(s): Mohamed Labouardy
  • Release date: October 2021
  • Publisher(s): Manning Publications
  • ISBN: 9781617297540