Chapter 1. Introduction to CloudForms

Welcome to this guide to mastering the Automate feature of Red Hat CloudForms. Before we begin our journey through Automate, it’s worth taking a general tour of CloudForms to establish a context for all that we’ll be learning.

What Is CloudForms?

Red Hat CloudForms is a cloud management platform that is also rather good at managing traditional server virtualization products such as VMware vSphere or Red Hat Enterprise Virtualization (RHEV). This broad capability makes it ideal as a hybrid cloud manager, able to manage both public clouds, and on-premises private clouds and virtual infrastructures. It provides a single management interface into a hybrid environment, enabling cross-platform orchestration to be achieved with relative simplicity.

Although originally a virtualization and Infrastructure as a Service (IaaS) cloud manager, CloudForms 4.0 introduced support for Docker container management, including Red Hat’s OpenShift Enterprise 3.x Platform as a Service (PaaS) cloud solution (see Figure 1-1).

Providers

CloudForms manages each cloud, container, or virtual environment using modular subcomponents called providers. Each provider contains the functionality required to connect to and monitor its specific target platform, and this provider specialization enables common cloud management functionality to be abstracted into the core product. In keeping with the manager of managers concept, CloudForms providers communicate with their respective underlying cloud or infrastructure platform using the native APIs published for the platform manager (such as VMware vCenter Server using the vSphere SOAP API).

The pluggable nature of the provider architecture makes it relatively straightforward to develop new providers to support additional cloud and infrastructure platforms. For example, the last two versions of CloudForms have added five new providers, with more currently under development.

Providers are broadly divided into categories, and with CloudForms 4.0 these are cloud, infrastructure, configuration management, and container.

Cloud Providers

CloudForms 4.0 ships with cloud providers that connect to and manage two public clouds: Amazon Web Services and Microsoft Azure. It also has a cloud provider that can connect to and manage a private or on-premises Red Hat OpenStack Platform (OSP) cloud (this is the OverCloud in the case that OSP is managed by the Red Hat OpenStack Platform 7 Director).

Infrastructure Providers

CloudForms 4.0 ships with infrastructure providers that connect to and manage VMware vCenter Server, Red Hat Enterprise Virtualization Manager, and Microsoft System Center Virtual Machine Manager. It also has an infrastructure provider that can connect to and manage a private or on-premises Red Hat OpenStack Platform 7 Director UnderCloud.

Configuration Management Providers

CloudForms 4.0 ships with a configuration management provider that can connect to and manage Red Hat Satellite 6. This enables CloudForms to import and use Satellite 6 host groups, and extends the provisioning capability to include bare-metal (i.e., nonvirtual) servers.

Container Providers

CloudForms 4.0 ships with container providers that can connect to and manage two Docker container managers: Red Hat Atomic Platform and Red Hat OpenShift Enterprise.

Mixing and Matching Providers

When deploying CloudForms in our enterprise we often connect to several providers. We can illustrate this with an example company.

Company XYZ Inc.

Our example organization has an established VMware vSphere 5.5 virtual environment, containing many hundreds of virtual machines. This virtual infrastructure is typically used for the stable, long-lived virtual machines, and many of the organization’s critical database, directory services, and file servers are situated here. Approximately half of the VMware virtual machines run Red Hat Enterprise Linux,¹ and to facilitate the patching, updating, and configuration management of these VMs, the organization has a Satellite 6 server.

Company XYZ is a large producer of several varieties of widget, and the widget development test cycle involves creating many short-lived instances in an OpenStack private cloud, to cycle through the test suites. The developers like to have a service catalog from which they can order one of the many widget test environments, and at any time there can be several hundred instances running various test configurations.

The web developers in the company are in the process of redeveloping the main Internet web portal as a scalable public cloud workload hosted in Amazon Web Services (AWS). The web portal provides a rich product catalog, online documentation, knowledge base, and ecommerce shopping area to customers.

In this organization, CloudForms manages the workflows that provision virtual machines into the vSphere virtual infrastructure, AWS, and OpenStack. The users have a self-service catalog to provision individual virtual machine workloads into either VMware or Amazon, or entire test suites into OpenStack. CloudForms orchestration workflows help with the maintenance of an image factory that keeps virtual machine images updated and published as VMware templates, Amazon Machine Images (AMIs), and OpenStack Glance images.

As part of the provisoning process CloudForms also manages the integration workflows that allow newly provisoned virtual machines to be automatically registered with the Satellite 6 server, and an in-house configuration management database (see Figure 1-2). This ensures that newly provisioned virtual machines are configured by Puppet according to server role and patched with the latest updates, with a full inventory visible to the help-desk system.

The Capabilities of CloudForms

We’ve already mentioned some of the capabilities of CloudForms such as orchestration, a service catalog, and integration workflows. Let’s have a look at the four main areas of capability: insight, control, automate, and integrate.

Insight

Insight is the process of gathering intelligence on our virtual or cloud infrastructure so that we can manage it effectively. It is one of the most fundamental but important capabilities of the product.

When we first connect a provider, CloudForms begins a process of discovery of the virtual or cloud infrastructure. An infrastructure provider will collect and maintain details of the entire virtual infrastructure, including clusters, hypervisors, datastores, virtual machines, and the relationships among them. Cloud vendors do not typically expose infrastructure details, so cloud providers will typically gather and monitor tenant-specific information on cloud components such as instances, images, availability zones, networks, and security groups.

CloudForms also stores and processes any real-time or historical performance data that the provider exposes. It uses the historical data to calculate useful trend-based analytics such as image or VM right-sizing suggestions and capacity-planning recommendations. It uses the real-time performance statistics and power-on/off events to give us insight into workload utilization and also uses this information to calculate metering and chargeback costs.

One of the roles of a CloudForms server is that of Smart Proxy. A server with this role has the ability to initiate a SmartState Analysis on a virtual machine, template, instance, or even Docker container. SmartState Analysis (also known as fleecing) is a patented technology that scans the container or virtual machine’s disk image to examine its contents. The scan discovers users and groups that have been added and applications that have been installed, and searches for and optionally retrieves the contents of specified configuration files or Windows Registry settings. This is an agentless operation that doesn’t require the virtual machine to be powered on.

CloudForms allows us to apply tags to infrastructure or cloud components to help us identify and classify our workloads or resources in a way that makes sense to our organization. These tags might specify an owning department, cost center, operating system type, location, or workload classification, for example. We can create powerful filters in the WebUI that allow us to display managed components such as VMs along organizational and business lines, rather than physical placement or characteristic.

To round off the summary of its insight ability, CloudForms also has a powerful reporting capability that can be used to create online or exportable CSV or PDF reports.

Control

We can use the Control functionality of CloudForms to enforce security and configuration policies, using the information retrieved from insight. For example, the SmartState Analysis of a virtual machine might discover a software package containing a known critical security vulnerability. We could implement a control policy to shut down the VM, or migrate it to a hypervisor in a quarantined network so that it can be patched.

Using real-time performance statistics, we might configure alerts to warn us when critical virtual machines are running at unusually high utilization levels. Many monitoring tools can do this, but with CloudForms we could also use such an alert to trigger an Automate workflow to dynamically scale out the application workload by provisioning more servers.

We can monitor for compliance with corporate security policies, by gathering and intelligently processing the contents of selected configuration files. In this way we might detect if SELinux has been disabled, for example, or that sshd is running with an insecure configuration. We can run such compliance rules automatically and mark a virtual machine as noncompliant, whereupon its status will be immediately visible in the WebUI.

Automate

One of the most powerful features of CloudForms is its ability to automate the orchestration of workloads and resources in our virtual infrastructure or cloud. Automate allows us to create and use powerful workflows using the Ruby scripting language and features provided by the Automation Engine, such as state machines and service models.

CloudForms comes preconfigured with a large number of out-of-the-box workflows, to orchestrate such things as:

Provisioning or scaling out of workloads, such as virtual machines or cloud instances
Provisioning or scaling out of infrastructure, such as bare-metal hypervisors or compute nodes
Scaling back or retirement of virtual machine or cloud instances

Each of these is done in the context of comprehensive role-based access control, with administrator-level approval of selected Automate operations required where appropriate.

We can extend or enhance these default workflows and create whole new orchestration workflows to meet our specific requirements.

Service catalog

We can create self-service catalogs to permit users to order our orchestration workflows with a single button click. CloudForms Automate comes with an interactive service dialog designer that we use to build rich dialogs, containing elements such as text boxes, radio buttons, or drop-down lists. These elements can be dynamically prepopulated with values that are specific and relevant to the logged-in user or workload being ordered.

Integrate

As an extension of its Automate capability, CloudForms is able to connect to and integrate with many enterprise tools and systems. The system comes with Ruby gems to enable automation scripts to connect to both RESTful and SOAP APIs, as well as libraries to connect to several SQL and LDAP databases, and the ability to run remote PowerShell scripts on Windows servers.

Typical integration actions might be to extend the virtual machine provisioning workflow to retrieve and use an IP address from a corporate IP address management (IPAM) solution; to create a new configuration item (CI) record in the central configuration management database (CMDB), or to create and update tickets in the enterprise service-management tool, such as ServiceNow.

The CloudForms Appliance

To simplify installation, CloudForms is distributed as a fully installed virtual machine, known as the CloudForms Management Engine (often referred to as appliance for convenience).

A CloudForms 4.0 appliance comes preconfigured with everything we need. It runs Red Hat Enterprise Linux 7.2, with PostgreSQL 9.4, Rails 4.2.5, the CloudForms application, and all associated Ruby gems installed. An appliance is downloadable as a virtual machine image template in formats suitable for VMware, Red Hat Enterprise Virtualization, OpenStack, or Microsoft System Center Virtual Machine Manager.

All software packages in a CloudForms appliance are installed from RPM files, just as with any other Red Hat Enterprise Linux server. The packages can be updated with yum update from a Satellite 6 server or the Red Hat content delivery network.

Ruby and Rails

CloudForms is a Ruby on Rails application that uses PostgreSQL as its database. When we use the Automate functionality of CloudForms, we work extensively with the Ruby language and write scripts that interact with a Ruby object model defined for us by the Automation Engine. We certainly don’t need to be Rails developers, however (we don’t really need to know anything about Rails), but as we’ll see in Chapter 6, some understanding of Rails concepts can make it easier to understand the object model and what happens behind the scenes when we run our scripts.

Note

Why Rails? Ruby on Rails is a powerful development framework for database-centric web-based applications. It is popular for open source product development; for example, Foreman, one of the core components of Red Hat’s Satellite 6.x product, is also a Rails application.

Projects, Products, and Some History

Red Hat is an open source company, and its products are derived from one or more “upstream” open source projects. In the case of CloudForms, the upstream project is ManageIQ. ²

ManageIQ (the Project)

The ManageIQ project releases a new version every six months (approximately). Each version is named alphabetically after a chess Grand Master, and so far these have been Anand, Botvinnik, and Capablanca. At the time of writing, Capablanca is the current release.

Red Hat CloudForms (the Product)

Red Hat CloudForms 1.0 was originally a suite of products comprising CloudForms System Engine, CloudForms Cloud Engine, and CloudForms Config Server, each with its own upstream project.

When Red Hat acquired ManageIQ (a privately held company) in late 2012, it decided to discontinue development of the original CloudForms 1.0 projects³ and base a new version, CloudForms 2.0, on the much more capable and mature ManageIQ Enterprise Virtualization Manager (EVM) 5.x product. EVM 5.1 was rebranded as CloudForms Management Engine 5.1.

It took Red Hat approximately 18 months from the time of the ManageIQ acquisition to make the source code ready to publish as an open source project. Once completed, the ManageIQ project was formed, and development was started on the Anand release.

CloudForms Management Engine (the Appliance)

CloudForms Management Engine is the name of the CloudForms virtual appliance that we download from redhat.com. The most recent versions of CloudForms Management Engine have been based on corresponding ManageIQ project releases. The relative versions and releases are summarized in Table 1-1.

Table 1-1. Summary of the relative project and product versions
ManageIQ project release	CloudForms Management Engine version	CloudForms version
	5.1	2.0
	5.2	3.0
Anand	5.3	3.1
Botvinnik	5.4	3.2
Capablanca	5.5	4.0

Summary

This chapter has introduced CloudForms at a fairly high level but has hopefully established a product context. The remainder of the book focuses specifically on the Automate functionality of CloudForms. Let’s roll up our sleeves and get started!

Mastering CloudForms Automation by Peter McGowan