DevOps Automation with Terraform and VMware

Chapter 1. Infrastructure as Code and Immutable Infrastructure Concepts

In this chapter, we explore concepts of operating an environment using Infrastructure as Code (IaC). You learn about immutable infrastructure and how to apply it to operational practices within your own VMware environment. We begin with a concept overview and learn the Terraform terminology. Next, we explore the Terraform vSphere provider and offer operational tips and guidance for your Terraform environment.

It is assumed that you have a version control system. Examples in this guide use Git with code hosted on GitHub.com in public repositories, which will be updated based on feedback and questions from readers.

Terraform Terminology and Component Descriptions

Let’s begin by defining some key phrases and terminology that you will encounter when using Terraform. These terms will show up throughout the guide and in the online documentation, and give context to what you are building in the example scenarios that we present.

Provider

This is your connector to the underlying infrastructure. This is how your declarative code will interact with the management API of whichever platform you are building on. The VMware vSphere provider will communicate with either vCenter for a full coverage or to a vSphere host, which provides less functionality.

Input variables

Input variables are passed to your Terraform configuration. These can be dynamically created or statically assigned. If not assigned programmatically, you will be prompted at the command-line interface (CLI) to enter values.

Data sources

These can be computed or queried infrastructure sources that are used in other parts of your Terraform configuration, such as clusters, resource pools, regions, or any of a variety of objects. Data sources will vary based on which provider you are using (e.g., Amazon Web Services [AWS], VMware, Digital Rebar).

Expressions

These are computed results that can range from literals to variables to indices, maps, and many other types. It’s possible to have queries that can also feed other expressions and do things like count within resources and then dynamically assign the count to naming resources. Another example is creating a virtual machine (VM) and then assigning the network interface on creation to a virtual switch or dynamically assigning VM names based on count, or environment name.

Functions

These are built-in functions that you can use in your expressions, including numeric (e.g., min, max), string (e.g., lower, upper, substr), collection (contains, flatten, sort, merge), and many others. You can find a full set of built-in functions here.

Output values

Return values from your Terraform environment. These can include static or dynamic results such as VM name, IP address, storage location, and other computed results that are available after a Terraform resource deployment or update. These are available programmatically from the CLI as soon as you have run your Terraform configuration.

Why DevOps and IaC?

Let’s begin by clearly describing what we mean by DevOps and IaC. There are many versions of the definition of DevOps. The often-referenced Wikipedia article describes it as the following:

DevOps is a set of software development practices that combines software development (Dev) and information technology operations (Ops) to shorten the systems development life cycle while delivering features, fixes, and updates frequently in close alignment with business objectives.

Similarly, the IaC definition is listed on Wikipedia as follows:

Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools.

The definition is less important than understanding the goal of these practices, which is to build and deploy infrastructure and applications in a consistent way, faster, and with less risk. It really is that simple.

Version control systems allow you to store, tag, and version your code centrally. The common “source of truth” of your code is then cloned or forked to the target client and servers, which ensures consistency. As each code and application update is stored, you can test it across environments (e.g., Dev, QA, Test, Production).

Using consistent build processes stored in code, in a collaborative way, ensures consistency of outcome and faster time to deploy. Using simple delegation of privileges for the application development teams allows the same deployment processes to be used across teams and across environments. Building application infrastructure will now be truly on-demand and consistent without the lengthy wait times to submit requests and hand tasks back and forth between teams.

Immutable Concepts and Capabilities with Terraform

Immutable infrastructure is a big shift from the traditional practice of build, deploy, patch, and maintain, which is the common method used in most data centers and even in cloud infrastructure today. Immutable infrastructure is built and deployed, and then designed to not be changed after deployment. The growth in features and popularity of containerized infrastructure has made immutable infrastructure practices simpler. VMs can be deployed in a similar way, provided that the application is built with immutable infrastructure in mind. The advantage to immutable infrastructure is the speed and stability when building and deploying applications, thanks to consistency of the underlying infrastructure layers.

Terraform allows for the rapid build, deployment, and tear down of infrastructure and applications. Figure 1-1 illustrates the flow as applications are developed and then built and deployed. Teams can choose to use packaging or build-from-source with repeatable deployment of the underlying infrastructure and the applications to ensure consistency across all environments.

Figure 1-1. Build and deployment cycle

Terraform Features of the VMware vSphere Provider

Each Terraform provider includes a number of data sources and parameters that you will use to configure your infrastructure. Each resource will have some minimum required values (e.g., name, cluster, host) and many optional parameters.

Should I Make Everything into IaC?

You probably have applications that receive limited updates, and it does not make sense to invest huge time and effort to codify into declarative resources. Use your judgement on whether each application or environment is likely to gain value from repeatable code-powered processes.

You can find configuration parameters and documentation for each data source on the Terraform website for the most updated version. Terraform is under continuous development and is rapidly expanding coverage. You can review information and check for updates in future versions for VMware providers and a growing list of other providers here.

Operational Practices for Terraform

Embracing IaC and DevOps methodologies also means changing some of your processes to adapt to this style of infrastructure management. This is a journey toward more agile, rapid, and consistent deployment of application resources with many stages of adoption. Don’t feel that you are losing the battle because you are not running 200 deploys a day like a LinkedIn or a Pinterest team would. Your goal is to take advantage of the power of Terraform and your existing VMware environment to move in the direction of IaC. As the proverb goes, “A journey of a thousand miles begins with a single step.”

Using Environment Variables for Configuration Parameters

Building and deploying applications and VMs with static server and location references is a risky practice. There is a chance that you will accidentally store content in the code repository, which will bleed into different environments when you deploy.

Storing configuration parameters as environment variables locally and on the remote application servers is particularly important with immutable infrastructure. As environments are spun up, they get configuration in memory. When changes are needed, modifications are also done in memory, which is quick and repeatable.

Simply define an environment variable such as TF_VAR_yourvariablename, as shown in Figure 1-2, and then you can refer to it anywhere in your Terraform configuration as ${var.your​variablename} for easy access and to remove the need for static parameters and constants defined in your code. Each server/instance can have its own environment to ensure that your configuration is localized and also dynamically created and modified.

Figure 1-2. Variables defined by environment type

Storing Secrets

This is the most contentious and challenging area when it comes to infrastructure operations. Where is the best place and product to store secrets (e.g., passwords, API keys, administrative network port information, Secure Shell [SSH] keys) for use in IaC?

Like the files and folders, the answer is, “It depends.” It’s ideal to use a secured, programmatically accessible secret storage platform (e.g., HashiCorp Vault, CyberArk, AWS Secrets Manager). The examples in this guide use locally stored credentials and secrets using environment variables on each system.

It’s critical that no secrets make their way into your code and into the repository. Even if you remove temporary passwords from code that has been previously committed to a repository, those previous versions can be searched and viewed by anyone with access to the code repository.

Terraform Process Flow

You should have a standard flow that you follow when using Terraform. Your flow will be to create a configuration, validate the code, check the live environment, run the configuration, and then check and potentially tear down the infrastructure later on. This directly relates to the common Terraform commands, as shown in Figure 1-3. Think of this as a resource life cycle.

Figure 1-3. Terraform process flow

There are other commands such as taint, untaint, refresh, and graph, which we touch on later in the guide, but these core commands will be the ones you encounter the most.

Why Terraform for VMware?

The ability to create declarative configurations without having to know and test all of the different APIs is significant. This also means that you might not need to build and understand complex scripts with multiple scripting languages that have been needed for automating VMware deployments up to now.

Terraform OSS is free and extensible for your other environments (e.g., Kubernetes, AWS, Microsoft Azure, Google Cloud Platform) with limited change required. Just swap out the provider and some configuration parameters, and you have the same declarative code for the rest of your hybrid environment.

Now that you have learned the fundamentals of IaC concepts and the core knowledge of your Terraform platform, it’s time to move on to working examples of the core features and functions of Terraform and the VMware vSphere. You can adapt these upcoming examples to your own local environment. They provide the foundation to begin applying these new concepts in your infrastructure and operations processes.