Deploying OpenStack

Chapter 4. Understanding Nova

Nova seeks to provide a framework for the large-scale provisioning and management of virtual compute instances. Similar in functionality and scope to Amazon’s EC2 service, it allows you to create, manage, and destroy virtual servers based on your own system images through a programmable API.

Nova Architecture

Nova is architected as a distributed application with many components, but the majority of these are custom-written Python daemons of two varieties:

• Web Server Gateway Interface (WSGI)^[2] applications to receive and mediate API calls

• Worker daemons to carry out orchestration tasks

However, there are two essential pieces of the architecture that are neither custom written nor Python-based: the messaging queue and the database. These two components facilitate the asynchronous orchestration of complex tasks through message passing and information sharing. Piecing this all together we get a picture like Figure 4-1.

Figure 4-1. Nova Logical Architecture

This complicated diagram can be summed up in three sentences:

• End users who want to use Nova to create compute instances call nova-api with OpenStack API or EC2 API requests.

• Nova daemons exchange information through the queue (actions) and database (information) to carry out these API requests.

• Glance is a completely separate service that Nova interfaces through the Glance API to provide virtual disk imaging services.

Now that we’ve seen the overview of the processes and their interactions, let’s take a closer look at each component.

API

The nova-api daemon is the heart of Nova. You may see it illustrated on many pictures of Nova as API and “Cloud Controller.” While this is partly true, cloud controller is really just a class (specifically the CloudController in nova/api/ec2/cloud.py) within the nova-api daemon. Its primary purpose is to accept and fulfill incoming API requests.

To accept and fulfill API requests, nova-api provides an endpoint for all API queries (accepting requests using either the OpenStack API or the Amazon EC2 API), initiates most of the orchestration activities (such as running an instance), and also enforces some policy (mostly quota checks). For some requests, it will fulfill the entire request itself by querying the database and then returning the answer. For more complicated requests, it will pass messages to other daemons through a combination of writing information to the database and adding messages to the queue.

By default, nova-api listens on port 8773 for the EC2 API and 8774 for the OpenStack API.

Scheduler

The nova-scheduler process is conceptually the simplest piece of code in Nova: it takes a virtual machine instance request from the queue and determines where it should run (specifically, which compute server host it should run on). In practice, however, this will grow to be the most complex piece, as it needs to factor in the current state of the entire cloud infrastructure and apply complicated algorithms to ensure efficient usage. To that end, nova-scheduler implements a pluggable architecture that lets you choose (or write) your own algorithm for scheduling. Table 4-1 details the current scheduler choices.

Table 4-1. Nova Schedulers

Scheduler	Notes
Simple	Attempts to find least loaded host.
Chance	Chooses random available host from service table. This is the default scheduler.
Zone	Picks random host from within an availability zone.

To illustrate how simple nova-scheduler can be, here is the relevant code from the chance scheduler class in nova/schedule/chance.py:

class ChanceScheduler(driver.Scheduler):
    """Implements Scheduler as a random node selector."""

    def schedule(self, context, topic, *_args, **_kwargs):
        """Picks a host that is up at random."""

        hosts = self.hosts_up(context, topic)
        if not hosts:
            raise driver.NoValidHost(_("Scheduler was unable to locate a host"
                                       " for this request. Is the appropriate"
                                       " service running?"))
        return hosts[int(random.random() * len(hosts))]

As you can see from above code sample, the schedule method simply chooses a random host from the array of hosts that are currently known to be “up.”

Compute Worker

The nova-compute process is primarily a worker daemon that creates and terminates virtual machine instances. The process by which it does so is fairly complex, but the basics are simple: accept actions from the queue and then perform one or a series of virtual machine API calls to carry them out while updating state in the database. An example of this would be nova-compute accepting a message from the queue to create a new instance and then using the libvirt library to start a new KVM instance.

There are a variety of ways that nova-compute manages virtual machines. The most common is through a software package called libvirt. This is a toolkit (API, daemon, and utilities) created by Red Hat to interact with the capabilities of a wide range of Linux virtualization technologies. While libvirt may be the most common, nova-compute also uses the Xen API, vSphere API, Windows Management Interface, and others to support other virtualization technologies.

One of strengths of Nova is its wide support for virtualization technologies. The virtualization technologies supported in the current release version of Nova are detailed in Table 4-2.

Table 4-2. Virtualization Support in Nova

Virtualization Product	Supported	Interface	Support Notes
Kernel Virtual Machine (KVM)	Yes	libvirt	Most popular technology for small scale deployments. Arguably the easiest to deploy and configure. Supports advanced operations such as live migration and resize.
Xen	Yes	libvirt	Most popular (along with XCP/XenServer) technology for larger scale and production deployments.
Citrix XenServer	Yes	XenAPI	Citrix’s commercial version of Xen-based virtualization product. Supports advanced features.
Xen Cloud Platform (XCP)	Yes	XenAPI	The open source version of Citrix XenServer available under the LGPL, GPL, Q Public License v1. Supports subset of XenServer features.
VMware ESX / ESXi / vSphere	Yes	vSphere API	Most popular enterprise virtualization platform. See the OpenStack VMware documentation for full information and restrictions when using this option.
VMware vSphere	No
User Mode Linux	Yes	libvirt	Generally considered a lower performance virtualization option, UML runs each guest as a regular process in user space.
Microsoft Hyper-V	Yes	Windows Management Instrumentation (WMI)	Hyper-V is Microsoft’s hypervisor-based virtualization technology.
QEMU	Yes	libvirt	Provides the basis for most Linux-based virtualization technologies (such as KVM and Virtualbox).
Linux Containers (LXC)	Yes	libvirt	LXC is an operating system-level partitioning technology that allows for running multiple isolated servers (containers) in a single kernel. LXC does not actually virtualize the server. Instead, it provides a virtual environment with its own process space. While this doesn’t provide the same level of isolation (as every partition shares the common kernel), it may provide some advantages in I/O performance.
Oracle VM VirtualBox	No

Virtualization Technology describes some important considerations that should be taken into account when choosing your virtualization technology.

Volume Worker

As you can gather by the name, nova-volume manages the creation, attaching, and detaching of persistent volumes to compute instances (similar in functionality to Amazon’s Elastic Block Storage). It can use volumes from a variety of providers such as iSCSI or AoE. Table 4-3 shows the current volume provider options.

Table 4-3. Nova Volume Provider Options

Volume Provider	Notes
AoE	High performance layer 2 Ethernet technology that encapsulates SATA commands in Ethernet frames. Supported on Linux through the AoE Tools package, specifically the vblade program.
iSCSI	A commonly used IP-based encapsulation of SCSI commands. This is supported by most modern operating systems, but the Nova implementation only currently supports Linux through with this implementation. This driver does support CHAP for authentication.
Solaris iSCSI	Supports Solaris-hosted iSCSI volumes and uses ZFS commands. Solaris server must be prepared by following the instructions in the nova/volume/san.py file.
Sheepdog	An open-source, distributed storage system specifically designed for QEMU/KVM installations that is developed by NTT Laboratories. More information is available at http://www.osrg.net/sheepdog/.
RBD	RADOS block device (RBD) driver to interact with Ceph, a distributed file system based on a reliable and scalable distributed object store. As stated on http://ceph.newdream.net/ the wiki, “Ceph is under heavy development, and is not yet suitable for any uses other than benchmarking and review.”
LeftHand	A driver for interacting with HP Lefthand SAN solutions (as of now known as “HP P4000 SAN Solutions”). Unlike other providers mentioned above, this provider does not run directly on the SAN hardware. Instead, it accesses it via SSH commands.

Network Worker

The nova-network worker daemon is very similar to nova-compute and nova-volume. It accepts networking tasks from the queue and then performs system commands to manipulate the network (such as setting up bridging interfaces or changing iptables rules).

Nova defines two different types of IP addresses for an instance: Fixed IPs and Floating IPs. These can be broadly thought of as private IPs (fixed) and public IPs (floating). Fixed IPs are assigned on instance startup and remain the same during their entire lifetimes. Floating IPs are dynamically allocated and associated to a domain to allow outside connectivity.

To support the assignment and connectivity of fixed IPs, Nova supports three networking managers:

• Flat is the most basic network manager. Each new instance is assigned a fixed IP address and attached to a common bridge (which must be created by the administrator). IP configuration information must be “injected” (written into the new instance virtual disk image) to configure the instance.

• FlatDHCP builds upon the Flat manager by providing DHCP services to handle instance addressing and creation of bridges.

• VLAN supports that most features. In this mode, nova-network creates a VLAN, a subnet, and a separate bridge for each project. Each project also receives a range of IP only accessible within the VLAN.

Of these three network managers, VLAN is the most featured, Flat is the most barebones (but flexible), and FlatDHCP strikes a nice balance between the two.

Queue

The queue provides a central hub for passing messages between daemons. This is currently implemented with RabbitMQ today, but theoretically could be any AMPQ message queue supported by the Python ampqlib and carrot libraries.

Nova creates several types of message queues to facilitate communication between the various daemons. These include: topics queues, fanout queues, and host queues. Topics queues allow messages to be broadcast to the number of particular class of worker daemons. For example, Nova uses these to pass messages to all (or any) of the compute or volume daemons. This allows Nova to use the first available worker to process the message. Host queues allow Nova to send messages to specific services on specific hosts. For example, Nova often needs to send a message to a specific host’s compute worker to take action on a particular instance. Fanout queues are only currently used for the advertising of the service capabilities to nova-scheduler workers.

Here is an example of the queues created in RabbitMQ for a simple all-in-one node installation:

$ sudo rabbitmqctl list_queues
Listing queues ...
scheduler_fanout_15b1731c5ac34aae8970369911f04542 0
volume    0
volume_fanout_e42438faedb84ab8aad8d85e29916424 0
compute_fanout_38a37d3dc7564b66a5a540a1e222b12b 0
compute.cactus 0
volume_fanout_d62eb016a76341f4899c91d5a8fbb0a9 0
volume_fanout_dcaebd5edb3045ff8b86636040d34071 0
volume.cactus 0
network_fanout_64b9cb80b2c34c7a8da983219c787147 0
compute    0
network_fanout_362393151e7c465a8e3ed003ac6dbc1b 0
compute_fanout_74165ee38c9d4c1ea1003ccd88a91c22 0
scheduler    0
network.cactus 0
network    0
scheduler_fanout_9444b4c8d5d6497b9b5e2df4eca33b0d 0
scheduler.cactus 0

As you can see from the example, topic, fanout, and host queues have been created for each service (nova-scheduler, nova-compute, nova-volume, nova-network).

Database

The database stores most of the configuration and run-time state for a cloud infrastructure. This includes the instance types that are available for use, instances in use, networks available, and projects. Table 4-4 details all the tables in the current Nova database scheme.

Table 4-4. Nova Database Schema

Table Name	Description
migrate_version	Stores current version of the database schema as well as other migration-related info. Only used internally and by developers during upgrades.
migrations	Used for running host-to-host migration.
auth_tokens	Maps Authorization tokens (for all API transactions) to actual users (via the user id field).
certificates	Mappings for user, projects, and x509 certificates files
networks	Information pertaining to networks defined in Nova. Includes IP addressing, VLAN, and VPN information.
compute_nodes	Capabilities (vcpus, memory, etc.) and state (vcpus used, memory used, etc.) of each compute node.
projects	Information about projects, including project manager.
console_pools	Pool of consoles on the same physical node.
quotas	Quota overrides for particular projects. This is discussed further in Quotas.
consoles	Console session for an instance.
export_devices	Shelf and blade information used primarily with the AoE volume driver.
security_group_rules, security_groups and security_group_instance_association	Represent security groups and their associated rules.
fixed_ips and floating_ips	Associates the fixed and floating IP addresses to instances.
services	Listing off registered services (nova-scheduler, nova-compute and so on) and their current state. The updated_at field is used to determine if a given service is considered healthy or not.
instance_actions	Lists guest VM’s actions and results.
user_project_association, user_project_role_association and user_role_association	Maintains relationship among users, projects, and roles.
instance_metadata	Metadata key/value pairs for an instance that is used during instance startup.
instance_types	Specifications (vCPUs, RAM, etc.) of flavors or instances types that users can use in this cloud. Described in much greater detail in Instance Types and Flavors.
instances	Representation of virtual machine instances.
users	Representations of users.
iscsi_targets	Mapping of iSCSI targets, hosts, and volumes.
volumes	Representation of volumes in the cloud.
key_pairs	Public key pairs for SSH.
zones	Represents a child zone of this zone. Only used in advanced configurations.

Nova supports a wide range of databases, including popular open-source stalwarts like MySQL and PostgreSQL. For more information on choosing an appropriate database for Nova, see Database.

---

^[2] WSGI is a Python standard specification that defines the communications between web and application servers.