O'Reilly logo

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Infrastructure as Code (IAC) Cookbook

Book Description

Over 90 practical, actionable recipes to automate, test, and manage your infrastructure quickly and effectively

About This Book

  • Bring down your delivery timeline from days to hours by treating your server configurations and VMs as code, just like you would with software code.
  • Take your existing knowledge and skill set with your existing tools (Puppet, Chef, or Docker) to the next level and solve IT infrastructure challenges.
  • Use practical recipes to use code to provision and deploy servers and applications and have greater control of your infrastructure.

Who This Book Is For

This book is for DevOps engineers and developers working in cross-functional teams or operations and would now switch to IAC to manage complex infrastructures.

What You Will Learn

  • Provision local and remote development environments with Vagrant
  • Automate production infrastructures with Terraform, Ansible and Cloud-init on AWS, OpenStack, Google Cloud, Digital Ocean, and more
  • Manage and test automated systems using Chef and Puppet
  • Build, ship, and debug optimized Docker containers
  • Explore the best practices to automate and test everything from cloud infrastructures to operating system configuration

In Detail

Infrastructure as Code (IAC) is a key aspect of the DevOps movement, and this book will show you how to transform the way you work with your infrastructure—by treating it as software.

This book is dedicated to helping you discover the essentials of infrastructure automation and its related practices; the over 90 organized practical solutions will demonstrate how to work with some of the very best tools and cloud solutions.

You will learn how to deploy repeatable infrastructures and services on AWS, OpenStack, Google Cloud, and Digital Ocean. You will see both Ansible and Terraform in action, manipulate the best bits from cloud-init to easily bootstrap instances, and simulate consistent environments locally or remotely using Vagrant. You will discover how to automate and test a range of system tasks using Chef or Puppet. You will also build, test, and debug various Docker containers having developers' interests in mind.

This book will help you to use the right tools, techniques, and approaches to deliver working solutions for today's modern infrastructure challenges.

Style and approach

This is a recipe-based book that allows you to venture into some of the most cutting-edge practices and techniques about IAC and solve immediate problems when trying to implement them.

Downloading the example code for this book. You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the code file.

Table of Contents

  1. Infrastructure as Code (IAC) Cookbook
    1. Table of Contents
    2. Infrastructure as Code (IAC) Cookbook
    3. Credits
    4. About the Authors
    5. About the Reviewer
    6. www.PacktPub.com
      1. eBooks, discount offers, and more
        1. Why Subscribe?
    7. Customer Feedback
    8. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Sections
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
        5. See also
      5. Conventions
      6. Reader feedback
      7. Customer support
        1. Downloading the example code
        2. Downloading the color images of this book
        3. Errata
        4. Piracy
        5. Questions
    9. 1. Vagrant Development Environments
      1. Introduction
      2. Adding an Ubuntu Xenial (16.04 LTS) Vagrant box
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
      3. Using a disposable Ubuntu Xenial (16.04) in seconds
        1. Getting ready
        2. How to do it…
        3. How it works…
      4. Enabling VirtualBox Guest Additions in Vagrant
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
      5. Using a disposable CentOS 7.x with VMware in seconds
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
        5. See also
      6. Extending the VMware VM capabilities
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
      7. Enabling multiprovider Vagrant environments
        1. Getting ready
        2. How to do it…
        3. How it works…
      8. Customizing a Vagrant VM
        1. Getting ready
        2. How to do it…
          1. Set the hostname
          2. Disable new box version check at startup
          3. Use a specific box version
          4. Display an informational message to the user
          5. Specify a minimum Vagrant version
      9. Using Docker with Vagrant
        1. Getting ready
        2. How to do it…
          1. Using NGINX Docker container through Vagrant
          2. Exposing Docker ports in Vagrant
          3. Sharing folders with Docker through Vagrant
        3. There's more…
      10. Using Docker in Vagrant for a Ghost blog behind NGINX
        1. Getting ready
        2. How to do it…
        3. There's more…
          1. A Docker Compose equivalent
      11. Using Vagrant remotely with AWS EC2 and Docker
        1. Getting ready
        2. How to do it…
      12. Simulating dynamic multiple host networking
        1. Getting ready
        2. How to do it…
        3. There's more…
          1. Speed up deployments with linked clones
          2. Using named NAT networks
      13. Simulating a networked three-tier architecture app with Vagrant
        1. Getting ready
        2. How to do it…
          1. Tier 3 – the database
          2. Tier 2: the application servers
            1. The Node.js application
          3. Tier 1: the NGINX reverse proxy
      14. Showing your work on the LAN while working with Laravel
        1. Getting ready
        2. How to do it…
          1. A sample NGINX configuration for Laravel
          2. Simple shell provisioning
          3. Enable provisioning
          4. Shared folder
          5. Public LAN Networking
        3. There's more…
      15. Sharing access to your Vagrant environment with the world
        1. Getting ready
        2. How to do it…
          1. Provisioning
          2. Starting Ghost engine
          3. Sharing access
            1. HTTP
          4. SSH
      16. Simulating Chef upgrades using Vagrant
        1. Getting ready
        2. How to do it…
          1. Vagrant Omnibus Chef plugin
          2. A sample Chef recipe
          3. Vagrant and Chef integration
          4. Testing the Chef version update
        3. There's more…
          1. Controlling default Vagrant VMs
          2. Berkshelf and Vagrant
          3. Testing with Test Kitchen
      17. Using Ansible with Vagrant to create a Docker host
        1. Getting ready
        2. How to do it…
          1. A simple Ansible Docker playbook for Vagrant
          2. Apply Ansible from Vagrant
        3. There's more…
      18. Using Docker containers on CoreOS with Vagrant
        1. Getting ready
        2. How to do it…
        3. There's more…
    10. 2. Provisioning IaaS with Terraform
      1. Introduction
      2. Configuring the Terraform AWS provider
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
      3. Creating and using an SSH key pair to use on AWS
        1. Getting ready
        2. How to do it…
        3. How it works…
        4. There's more…
      4. Using AWS security groups with Terraform
        1. Getting ready
        2. How to do it…
        3. There's more…
      5. Creating an Ubuntu EC2 instance with Terraform
        1. Getting ready
        2. How to do it…
          1. Scaling the number of instances
        3. There's more…
      6. Generating meaningful outputs with Terraform
        1. Getting ready
        2. How to do it…
        3. There's more…
      7. Using contextual defaults with Terraform
        1. Getting ready
        2. How to do it…
        3. There's more…
      8. Managing S3 storage with Terraform
        1. Getting ready
        2. How to do it…
        3. There's more…
      9. Creating private Docker repositories with Terraform
        1. Getting ready
        2. How to do it…
      10. Creating a PostgreSQL RDS database with Terraform
        1. Getting ready
        2. How to do it…
        3. There's more…
      11. Enabling CloudWatch Logs for Docker with Terraform
        1. Getting ready
        2. How to do it…
          1. Amazon CloudWatch Logs Docker logging driver
      12. Managing IAM users with Terraform
        1. Getting ready
        2. How to do it…
          1. An IAM user for S3 access
          2. Testing the restrictions
          3. An IAM user for EC2 in read-only
          4. An application user IAM – CloudWatch Logs
        3. There's more…
    11. 3. Going Further with Terraform
      1. Introduction
      2. Handling different environments with Terraform
        1. Getting ready
        2. How to do it…
          1. Keeping the tfstate isolated
          2. Setting the production flag
      3. Provisioning a CentOS 7 EC2 instance with Chef using Terraform
        1. Getting ready
        2. How to do it…
          1. Creating the EC2 instance
          2. Passing connection information
          3. Giving Chef information
        3. How it works…
        4. There's more…
      4. Using data sources, templates, and local execution
        1. Getting ready
        2. How to do it…
          1. Data and templates
          2. The local-exec Terraform provisioner
          3. Apply a configured Ansible
      5. Executing remote commands at bootstrap using Terraform
        1. Getting ready
        2. How to do it…
      6. Using Docker with Terraform
        1. Getting ready
        2. How to do it…
      7. Simulating infrastructure changes using Terraform
        1. Getting ready
        2. How to do it…
          1. Planning
          2. Quickly simulating changes
          3. Targeting for a specific change
      8. Teamwork – sharing Terraform infrastructure state
        1. Getting ready
        2. How to do it…
          1. Sharing with Git
          2. Sharing remotely with S3
          3. Sharing remotely with Consul
          4. Other state sharing options
      9. Maintaining a clean and standardized Terraform code
        1. Getting ready
        2. How to do it…
          1. Syntax validation
          2. Style validation
      10. One Makefile to rule them all
        1. Getting ready
        2. How to do it…
        3. See also
      11. Team workflow example
        1. Getting ready
        2. How to do it…
          1. A simple Git repository
          2. Initial infrastructure code
          3. Terraform code validation
          4. Infrastructure code commit
          5. Make a pull request
          6. Apply the changes
      12. Managing GitHub with Terraform
        1. Getting ready
        2. How to do it…
          1. Configuring GitHub
          2. Adding users to the GitHub organization
          3. Adding GitHub teams
          4. Setting Git repository access rights
      13. External monitoring integration with StatusCake
        1. Getting ready
        2. How to do it…
          1. Creating an automated ping monitoring test
          2. Creating an HTTPS test
    12. 4. Automating Complete Infrastructures with Terraform
      1. Introduction
      2. Provisioning a complete CoreOS infrastructure on Digital Ocean with Terraform
        1. Getting ready
        2. How to do it…
          1. Handling the SSH key
          2. Creating the CoreOS cluster members
          3. Adding useful output
          4. Dynamic DNS Integration
          5. Integrating cloud-init
          6. Integrating dynamic StatusCake monitoring
      3. Provisioning a three-tier infrastructure on Google Compute Engine
        1. Getting ready
        2. How to do it…
          1. Generating API credentials for a Google project
          2. Creating Google Compute HTTP instances
          3. Creating a Google Compute Firewall rule
          4. Load balancing Google Compute instances
          5. Creating a Google MySQL database instance
          6. Adding some useful outputs
      4. Provisioning a GitLab CE + CI runners on OpenStack
        1. Getting ready
        2. How to do it…
          1. Configuring the OpenStack provider
          2. Creating a key pair on OpenStack
          3. Creating a security group on OpenStack
          4. Creating block storage volumes on OpenStack
          5. Creating compute instances on OpenStack
          6. Creating an object storage container on OpenStack
          7. Applying
      5. Managing Heroku apps and add-ons using Terraform
        1. Getting ready
        2. How to do it…
          1. Creating a Heroku application with Terraform
          2. Adding Heroku add-ons using Terraform
          3. Using Heroku with Terraform
      6. Creating a scalable Docker Swarm cluster on bare metal with Packet
        1. Getting ready
        2. How to do it…
          1. Creating a Packet project using Terraform
          2. Handling Packet SSH keys using Terraform
          3. Bootstraping a Docker Swarm manager on Packet using Terraform
          4. Bootstraping Docker Swarm nodes on Packet using Terraform
          5. Using the Docker Swarm cluster
    13. 5. Provisioning the Last Mile with Cloud-Init
      1. Introduction
      2. Using cloud-init on AWS, Digital Ocean, or OpenStack
        1. Getting ready
        2. How to do it…
          1. Using cloud-init on Amazon Web Services
          2. Using cloud-init on Digital Ocean
          3. Using cloud-init on OpenStack
          4. Combining cloud-init and Terraform for any IaaS
      3. Handling files using cloud-init
        1. Getting ready
        2. How to do it…
      4. Configuring the server's time zone using cloud-init
        1. Getting ready
        2. How to do it…
      5. Managing users, keys, and credentials using cloud-init
        1. Getting ready
        2. How to do it…
      6. Managing repositories and packages using cloud-init
        1. Getting ready
        2. How to do it…
      7. Running commands during boot using cloud-init
        1. Getting ready
        2. How to do it…
      8. Configuring CoreOS using cloud-init
        1. Getting ready
        2. How to do it…
          1. Configuring etcd using cloud-init
          2. Configuring fleet using cloud-init
          3. Configuring the update strategy using cloud-init
          4. Configuring locksmith using cloud-init
          5. Configuring systemd units using cloud-init
          6. Configuring flannel using cloud-init
      9. Deploying Chef Client from start to finish using cloud-init
        1. Getting ready
        2. How to do it…
          1. Deploying the Chef omnibus installer using cloud-init
          2. Configuring Chef against a Chef Server organization using cloud-init
          3. Applying a Chef cookbook at bootstrap using cloud-init
      10. Deploying a remote Docker server using cloud-init
        1. Getting ready
        2. How to do it...
          1. Setting the timezone on CoreOS using cloud-init
          2. Enabling Docker TCP socket for network access
        3. There's more...
        4. See also
    14. 6. Fundamentals of Managing Servers with Chef and Puppet
      1. Introduction
      2. Getting started (notions and tools)
          1. Running Chef
          2. Chef plugins
          3. Chef organizations
          4. Chef nodes
          5. Chef environments
          6. Chef roles
          7. Chef resources
          8. Chef recipes
          9. Chef cookbooks
          10. Chef run list
        1. There's more…
      3. Installing the Chef Development kit and Puppet Collections
        1. Getting ready
        2. How to do it…
          1. Chef DK contents
        3. How it works…
        4. There's more…
        5. See also
      4. Creating a free hosted server Chef account and a Puppet server
        1. Getting ready
        2. How to do it…
        3. There's more…
      5. Automatically bootstrapping a Chef client and a Puppet agent
        1. Getting ready
        2. How to do it…
        3. There's more…
      6. Installing packages
        1. Getting ready
        2. How to do it…
          1. Generating an empty Apache cookbook
          2. Uploading the cookbook
          3. Applying the cookbook
          4. Creating a MariaDB cookbook
          5. Creating a PHP cookbook
        3. There's more…
        4. See also
      7. Managing services
        1. Getting ready
        2. How to do it…
          1. Enabling and starting Apache service
          2. Enabling and starting the MariaDB service
        3. There's more…
        4. See also
      8. Managing files, directories, and templates
        1. Getting ready
        2. How to do it…
          1. Managing a simple static file
          2. Managing dynamic files and directories from a template
        3. There's more…
        4. See also
      9. Handling dependencies
        1. Getting ready
        2. How to do it…
        3. There's more…
        4. See also
      10. More dynamic code using notifications
        1. Getting ready
        2. How to do it…
        3. There's more…
        4. See also
      11. Centrally sharing data using a Chef data bag and Hiera with Puppet
        1. Getting ready
        2. How to do it…
        3. There's more…
        4. See also
      12. Creating functional roles
        1. Getting ready
        2. How to do it…
        3. There's more…
        4. See also
      13. Managing external Chef cookbooks and Puppet modules
        1. Getting ready
        2. How to do it…
          1. Using the official MySQL cookbook and its dependencies with Berkshelf
          2. Including dependencies in a role
          3. Uploading cookbook dependencies using Berkshelf
          4. Testing MySQL deployment
        3. There's more…
        4. See also
    15. 7. Testing and Writing Better Infrastructure Code with Chef and Puppet
      1. Introduction
      2. Linting Chef code with Foodcritic and Puppet code with puppet-lint
        1. Getting ready
        2. How to do it…
          1. Cookstyle
          2. Foodcritic
        3. There's more…
          1. Puppet coding style
          2. Documentation
        4. See also
      3. Unit testing with ChefSpec and rspec-puppet
        1. Getting ready
        2. How to do it…
          1. The Spec Helper
          2. Testing a successful Chef run context
          3. Testing a package installation
          4. Testing services status
          5. Testing another recipe from the same cookbook
          6. Testing directory creation
          7. Testing file creation
          8. Testing templates creation
          9. Stubbing data bags for searches
          10. Testing recipes inclusion
          11. Intercepting errors in tests
        3. There's more…
        4. See also
      4. Testing infrastructure with Test Kitchen for Chef and Beaker for Puppet
        1. Getting ready
        2. How to do it…
          1. Configuring Test Kitchen
          2. Testing with Test Kitchen
        3. How it works…
        4. There's more…
        5. See also
      5. Integration testing with ServerSpec
        1. Getting ready
        2. How to do it…
          1. Creating a ServerSpec helper script
          2. Testing a package installation
          3. Testing for service status
          4. Testing for listening ports
          5. Testing for files existence and content
          6. Testing for repository existence
        3. There's more…
        4. See also
    16. 8. Maintaining Systems Using Chef and Puppet
      1. Introduction
      2. Maintaining consistent systems using scheduled convergence
        1. Getting ready
        2. How to do it…
          1. Using the Chef client as a daemon
          2. Tweaking the convergence interval time
          3. Running the Chef client as a cron
          4. Tweaking the Chef cron job
        3. There's more…
        4. See also
      3. Creating environments
        1. Getting ready
        2. How to do it…
          1. Creating a production environment
          2. Setting an environment to a node
          3. Bootstrapping a node with an environment
          4. Fixing cookbook versions for an environment
          5. Overriding attributes for an environment
          6. Accessing the environment from a recipe
        3. There's more...
          1. Manual environment creation in the Puppet server
          2. Node environment selection
          3. Getting the environment from manifests
          4. The dynamic way – r10k
        4. See also
      4. Using Chef encrypted data bags and Hiera-eyaml with Puppet
        1. Getting ready
        2. How to do it…
          1. Encrypting data bags with a shared secret
          2. Accessing an encrypted data bag in the CLI
          3. Using an encrypted data bag from a recipe
        3. There's more…
          1. Preparing the Puppet server
          2. Preparing the workstation
          3. Securing the MySQL root password
        4. See also
      5. Using Chef Vault encryption
        1. Getting ready
        2. How to do it…
          1. Accessing the encrypted vault from a cookbook
        3. See also
      6. Accessing and manipulating system information with Ohai
        1. Getting ready
        2. How to do it…
          1. Accessing Ohai information from a Chef recipe
        3. There's more…
        4. See also
      7. Automating application deployment (a WordPress example)
        1. Getting ready
        2. How to do it…
          1. Including dependencies
          2. Creating the application's database
          3. Deploying an application from git or GitHub
        3. There's more…
        4. See also
      8. Using a TDD workflow
        1. Getting ready
        2. How to do it…
          1. Infrastructure TDD – writing tests first
          2. Deploying Docker with Chef
          3. Linting the code
          4. Supporting another platform
          5. Team working using Chef and git
          6. Deploying to staging
          7. Deploying to production
        3. There's more…
        4. See also
      9. Planning for the worse – train to rebuild working systems
        1. Getting ready
        2. How to do it…
          1. Multi-machine recovery
        3. There's more…
    17. 9. Working with Docker
      1. Introduction
      2. Docker usage overview
        1. Getting ready
        2. How to do it…
          1. Running Bash in an Ubuntu 16.04 container
          2. Running Nginx in a container
          3. Sharing data with a container
          4. Building a container with utilities
          5. Using a private registry
        3. See also
      3. Choosing the right Docker base image
        1. Getting ready
        2. How to do it…
          1. Starting from an Ubuntu image
          2. Starting from a CentOS image
          3. Starting from a Red Hat Enterprise Linux (RHEL) image
          4. Starting from a Fedora image
          5. Starting from an Alpine Linux image
          6. Starting from a Debian image
          7. Linux distributions container image size table
          8. Starting from a Node JS image
          9. Starting from a Golang image
          10. Starting from a Ruby image
          11. Starting from a Python image
          12. Starting from a Java image
          13. Starting from a PHP image
        3. See also
      4. Optimizing the Docker image size
        1. Getting ready
        2. How to do it…
        3. How it works…
      5. Versioning Docker images with tags
        1. Getting ready
        2. How to do it…
      6. Deploying a Ruby-on-Rails web application in Docker
        1. Getting ready
        2. How to do it…
      7. Building and using Golang applications with Docker
        1. Getting ready
        2. How to do it…
          1. Using the golang Docker image to cross-compile a Go program
          2. Using the golang Docker image to build and ship a Go program
          3. Using the scratch Docker image
          4. Using the Alpine Linux alternative for a Go program
      8. Networking with Docker
        1. Getting ready
        2. How to do it…
          1. Docker networks
          2. Connecting multiple networks for one container
      9. Creating more dynamic containers
        1. Getting ready
        2. How to do it…
      10. Auto-configuring dynamic containers
        1. Getting ready
        2. How to do it…
      11. Better security with unprivileged users
        1. Getting ready
        2. How to do it…
      12. Orchestrating with Docker Compose
        1. Getting ready
        2. How to do it…
          1. Extending Docker Compose
        3. See also
      13. Linting a Dockerfile
        1. Getting ready
        2. How to do it…
          1. Hadolint
          2. Dockerfile_lint
      14. Deploying a private Docker registry with S3 storage
        1. Getting ready
        2. How to do it…
          1. Using an S3 backend
        3. See also
    18. 10. Maintaining Docker Containers
      1. Introduction
      2. Testing Docker containers with BATS
        1. Getting ready
        2. How to do it…
          1. Creating BATS tests
          2. Using Makefile to glue it all together
        3. See also
      3. Test-Driven Development (TDD) with Docker and ServerSpec
        1. Getting ready
        2. How to do it…
          1. Creating a ServerSpec environment using Bundler
          2. Initializing the tests
          3. TDD – using the Debian Jessie base's Docker image
          4. TDD – installing the NGINX package
          5. TDD – running NGINX
        3. See also
      4. The workflow for creating automated Docker builds from Git
        1. Getting ready
        2. How to do it…
          1. Creating an automated build on the Docker Hub
          2. Configuring a GitHub to a Docker Hub-automated build pipeline
          3. Building Docker images using Git tags
      5. The workflow for connecting the Continuous Integration (CI) system
        1. Getting ready
        2. How to do it…
      6. Scanning for vulnerabilities with Quay.io and Docker Cloud
        1. Getting ready
        2. How to do it…
          1. Using Docker Security Scanning
        3. How it works…
        4. See also
      7. Sending Docker logs to AWS CloudWatch logs
        1. Getting ready
        2. How to do it…
          1. Using the Docker run
          2. Using docker-compose
          3. Using systemd
        3. There's more...
      8. Monitoring and getting information out of Docker
        1. Getting ready
        2. How to do it...
          1. Using docker stats
          2. Using Google's cAdvisor tool
        3. See also
      9. Debugging containers using sysdig
        1. Getting ready
        2. How to do it...
        3. See also
    19. Index