Chapter 4. Building Your First Graphite Server
I hope I haven’t scared you away with all of the mental preparation and cautionary mumbo jumbo. Graphite really is a fun and effective tool. But like any complex piece of machinery (or software), it takes an experienced operator to configure and maintain it for running at peak efficiency. The previous chapters should have imbued you with enough practical knowledge around the Graphite ecosystem to help you clear most hurdles head-on.
This chapter (finally) explains the recommended procedure for installing and configuring your own Linux instance running Graphite, Carbon, Whisper, and the database where we’ll save our graphs and dashboards later. We’ll also cover a variety of housekeeping tasks, including web access controls (authentication and authorization), fixing permissions for the storage and logging directories, and, finally, testing that everything works as expected.
Note
Note that you should be able to run all of the Graphite components on any modern UNIX or UNIX-like system, including Linux, Solaris, and the family of BSD operating systems. We only cover Linux since it is the clear leader in terms of numbers of Graphite installations, and to keep this chapter from becoming even more unwieldy than it already is.
If you’re the kind of person who prefers to learn by diving in with both feet, I have good news for you. The Synthesize project was designed to help users quickly and painlessly launch a Graphite instance. We’ll get started immediately with Synthesize in the next section for anyone in a hurry. Feel free to jump ahead to “What Installation Methods Are Available?” if you’re more interested in the traditional installation methods.
Quick Start with Synthesize
There’s a good chance you’re already familiar with virtualization (and containerization) tools like Vagrant and Docker. While the latter is an enormously popular tool for deploying software using Linux containers (LXC), it’s rather a pain to use with Graphite due to the quantity and variety of executable processes and interconnected network services.
Fortunately, Vagrant fits the bill nicely, allowing us to build and share a virtual system configuration with our choice of Linux distribution, forward ports for host-to-guest communication (StatsD, Carbon, and Graphite-Web services), and include a provisioning script (for the actual installation process). This is the formula I chose for Synthesize, an open source project aimed at beginners who want a simple way to get up and running with Graphite.
Running Synthesize in Vagrant
Launching a Graphite virtual machine via Synthesize takes just a few easy steps. The following assumes you already have Git and Vagrant installed on your personal laptop or workstation:
$ git clone https://github.com/obfuscurity/synthesize.git $ cd synthesize $ vagrant up
Running Synthesize manually
If you’d rather use Synthesize to install Graphite on a “real” system (i.e., not a VM running on your laptop), you can do that too. The only limitation is that it must be running the supported base operating system required by Synthesize. At the time of this writing, that’s Ubuntu 14.04 LTS:
$ ssh server $ git clone https://github.com/obfuscurity/synthesize.git $ cd synthesize $ sudo ./install
Next steps with Synthesize
Once Synthesize is running, you’ll have a finely tuned Graphite instance running the various Graphite components, a high-performance StatsD server implementation (Statsite), and collectd. The collectd daemon has been specially configured to gather a variety of health and performance metrics for the Graphite and Carbon services on the instance.
For a full breakdown of the various service ports, web credentials, uninstallation methods, and more, I urge you to visit the project website for the most up-to-date version of this information.
If you chose the shortest path with Synthesize, I won’t be offended if you decide to jump ahead to the next chapter and start learning how to create graphs. On the other hand, you’re welcome to hang around and see how the sausage is made, so to speak.
Where Does Graphite Store All My Files?
No matter which method you choose, Graphite installs all files in the /opt/graphite prefix by default (see Example 4-1). Distributions that package Graphite for their users often choose a directory layout compliant with the filesystem hierarchy standard (FHS).
I respect your choice to put your files where it makes sense for you and your business, but for my money I prefer my files in the default location. It’s easier to back everything up in one fell swoop, and the Graphite community will be better informed to help you out in case something goes sideways.
Nevertheless, many of the subdirectories can be changed in your configuration settings. Here are some of the more important directories and a brief description of what they contain.
GRAPHITE_ROOT-
The installation root for Graphite. Defaults to
/opt/graphite. CONF_DIR-
All Carbon-related configuration files are stored here. It’s common to find some Graphite-Web settings files here, as well (dashboard.conf, graphTemplates.conf, etc), although those can be overridden in Graphite’s local_settings.py file (discussed later). Inherits from
GRAPHITE_CONF_DIRif that environment variable is set. Defaults toGRAPHITE_ROOT/conf. STORAGE_DIR-
The root for all storage-related files, including Whisper and RRD databases, logs and pidfiles. Inherits from
GRAPHITE_STORAGE_DIRif that environment variable is set. Defaults to/opt/graphite/storage. LOCAL_DATA_DIR-
Tells Carbon where to write your Whisper files. Defaults to
STORAGE_DIR/whisper. WHISPER_DIR-
Not to be confused with
LOCAL_DATA_DIR(although they should be the same value), this setting tells Graphite-Web where to find all the Whisper files to read from. Defaults to/opt/graphite/storage/whisper. RRD_DIR-
Where Graphite-Web should look for RRD database files. Defaults to
/opt/graphite/storage/rrd. LOG_DIR-
The directory containing logs for each of the Carbon daemons and the Graphite web application (
webapp). Defaults toSTORAGE_DIR/log. PID_DIR-
The directory where Carbon daemons write their pidfiles. Defaults to
STORAGE_DIR.
Example 4-1. Typical Graphite directory structure
/opt/graphite
├── /opt/graphite/bin
├── /opt/graphite/conf
├── /opt/graphite/lib
├── /opt/graphite/storage
│ ├── /opt/graphite/storage/log
│ ├── /opt/graphite/storage/rrd
│ └── /opt/graphite/storage/whisper
└── /opt/graphite/webapp
├── /opt/graphite/webapp/content
└── /opt/graphite/webapp/graphiteAre Packages Available for My Distro?
If you’re using one of the more popular Linux distributions, there’s a good chance they’re already building binary packages of the various Graphite components. Although you’re certainly welcome to use those, I urge you to at least skim through the rest of this section so you have a better understanding of Graphite’s software dependencies and installation process.
What Installation Methods Are Available?
The Graphite project officially supports two different installation methods: binary packages hosted on the Python Package Index (PyPI) and installed using the pip package manager, and manual installation from the project source code using the respective setup.py script for each component.
You might think pip packages are the way to go, and for many folks they’re a good choice. However, I personally prefer installing from source due to the flexibility you gain—not in terms of installations or upgrades so much, but rather for the convenience of testing new branches and bug fixes. In addition, having a local checkout makes it that much easier to contribute new Graphite-Web rendering functions back upstream. If you don’t believe me, take a look at the list of contributors for Graphite-Web’s functions.py sometime:
$ git shortlog -s webapp/graphite/render/functions.py | wc -l 63
Because I’m the one writing this book, I get the pleasure of informing you that we’re going with the source installation method. Never fear—it’s actually very simple and straightforward. The software dependencies are the same either way, so the difference between pip and source installs is minimal.
Should I Use Virtualenv?
Virtualenv is a tool for creating isolated Python environments. It’s commonly used by Python developers to build small, self-contained environments within each project’s development folder. When you use (and activate) virtualenv, all Python binaries and libraries needed for that project are copied into the designated folder structure. This self-containment makes it possible to run countless different versions of the Python interpreter and third-party libraries on the same system, each safe from interaction with the others.
The question remains: should you use virtualenv with your Graphite installation?
The obvious answer, in my opinion, is a resounding yes. Graphite’s components have a nontrivial number of Python dependencies, many of them “locked” to a specific version number. Being able to contain these within the Graphite directory structure means you’re reasonably protected from changes made to your system-wide Python toolchain. The only caveat is to remember to adjust your PATH to always load your Graphite bin directory ahead of your other paths. Don’t worry—I’ll remind you when it’s time.
Note
Graphite versions 0.9.15 and older are hardcoded to install to /opt/graphite. This will change in the next major release. Until then, however, it means that although we can use virtualenv, it must use /opt/graphite as our base installation path. If you wish to try out a development version of Graphite, you can use virtualenv to install Graphite anywhere, including your own home directory.
Using sudo Effectively
Throughout the following examples, you’ll notice that I run any privileged commands (where we need root permissions) using the sudo command. This is a widely accepted best practice, preferred over logging in as the superuser and running the commands in a full login shell. However, in order to maintain a high level of security, the default sudo configuration often disables the ability to “pass through” environment variables (such as PATH) to the intended user.
Because we’re using virtualenv, it’s vital we maintain our custom PATH to ensure we’re using the intended python and pip binaries. We can take a couple quick steps to enable this behavior. The first is to disable the secure_path option in your /etc/sudoers file (edit with sudo visudo). The next is to run any affected sudo commands with the -E argument, which preserves the existing user environment variables.
Your business or organization may have policies that restrict these sorts of changes to your sudo configuration. Fortunately there are a variety of methods you can use to achieve the intended goal: installing Graphite in a system-wide location (/opt/graphite) and setting our PATH to work properly with virtualenv. You can follow the examples I’ve provided or you can adapt your own best practices to meet these requirements.
Dependencies
There are a handful of software dependencies we’ll need to install before we get to the actual Graphite bits. Unlike the rest of our Graphite files and Python libraries, these will be installed from the operating system vendor’s package repository.
Debian/Ubuntu
Installing the packages on modern Debian or Ubuntu systems is straightforward. They already provide all of the dependencies we need in the official Deb repositories:
$ sudo apt-get install -y libcairo2-dev \
libffi-dev \
pkg-config \
python-dev \
python-pip \
virtualenv \
fontconfig \
apache2 \
libapache2-mod-wsgi \
memcached \
librrd-dev \
libldap2-dev \
libsasl2-dev \
git-core \
gcc
RHEL/Fedora/CentOS
The dependencies for RPM-based systems are similar, with one exception. We’ll first need to install the Extra Packages for Enterprise Linux (EPEL) repository metadata where the python-pip package is distributed:
$ sudo yum install -y epel-release
$ sudo yum install -y cairo-devel \
libffi-devel \
pkgconfig \
python-devel \
python-pip \
python-virtualenv \
fontconfig \
httpd \
mod_wsgi \
memcached \
rrdtool-devel \
openldap-devel \
cyrus-sasl-devel \
git \
gcc
Installing from Source
With the system-level dependencies complete, we can move on to the fun stuff. We’re going to start by creating a virtualenv environment at /opt/graphite where all of our components will be installed. Once this is ready, we’ll clone each of the projects from their GitHub repositories to the local filesystem, select the version we want to install with git checkout, and then install each component with its setup.py script.
Suffice it to say you need to run each installation step in the order in which I am about to describe them. However, as long as you create your virtualenv first, you can always safely rm -rf /opt/graphite and start over from scratch if something goes awry.
Note
Many of the following commands will log a bunch of output to the screen. Under normal circumstances, this content can be safely ignored. If there’s a serious problem, the command should die and provide some errors or warnings that can be useful in searching for answers online. For the sake of brevity, I’m only listing some of the available commands.
First thing first, we need to create our working environment where the Graphite components and their Python dependencies will reside:
$ sudo virtualenv /opt/graphite $ export PATH=/opt/graphite/bin:$PATH
Next we’ll clone the Whisper repository, check out the desired version, and run the installation script. We install Whisper first because it also happens to be a dependency for Carbon and Graphite-Web:
$ cd /usr/local/src $ sudo git clone https://github.com/graphite-project/whisper.git $ cd whisper $ sudo git checkout 0.9.15 $ sudo -E python setup.py install
Logically it makes sense to handle Carbon next, although it really doesn’t matter. Note the appearance of the pip install -r command, which resolves all Python dependencies listed in requirements.txt. Everything else is the same process as before:
$ cd /usr/local/src $ sudo git clone https://github.com/graphite-project/carbon.git $ cd carbon $ sudo git checkout 0.9.15 $ sudo -E pip install -r requirements.txt $ sudo -E python setup.py install
Finally, we install Graphite-Web using the same method. There are a couple of new commands here to handle optional dependencies for rrdtool and python-ldap. If you don’t need to read from RRD files or use LDAP authentication, you can safely skip that command. The check-dependencies.py script is handy for testing that our Python dependencies installed successfully. If you choose to skip the RRD and LDAP packages, you’ll see warnings, but they should be marked as OPTIONAL and can be safely ignored:
$ cd /usr/local/src $ sudo git clone https://github.com/graphite-project/graphite-web.git $ cd graphite-web $ sudo git checkout 0.9.15 $ sudo -E pip install -r requirements.txt $ sudo -E pip install rrdtool python-ldap $ python check-dependencies.py $ sudo -E python setup.py install
At this point, you’re effectively done installing Graphite. Congratulations! We still have a little ways to go before your Graphite server is ready to use, but the “scary” part is behind us.
Of course, if you chose to run the Synthesize project, you’d already be collecting metrics and graphing charts by now. That’s OK, I still heart you. Let’s move on to setting up our web database.
Preparing Your Web Database
Graphite-Web uses a relational database to store graph and dashboard configuration, user accounts (if you choose to authenticate against the database), and even Graphite events data. As I mentioned in the previous chapter, SQLite is not a good choice for even moderately busy production Graphite services, but it’s fine for development and test systems.
Note
PostgreSQL fans need not worry—we’ll cover the use of PostgreSQL for high-availability clusters in Chapter 9. But for now, we’re going to focus on getting up and running with SQLite.
Django uses a database schema format known as fixtures. Graphite-Web includes one of these fixtures that holds all of the data and relationships it needs to support the aforementioned feature set.
Running the django-admin.py syncdb command as shown in Example 4-2 will install the fixtures and also prompt you to create a superuser account. You’ll want to do this now so you can administer your web database later. You should also see a warning about setting your SECRET_KEY. This is an important security step we’ll address later when we configure Graphite-Web’s local_settings.py file.
Warning
The syncdb subcommand for django-admin.py was deprecated in Django version 1.7, superseded by migrate. We’ll continue to use the former throughout this book since most users are probably still running on an older version of Django. If you’re running Django version 1.9 or newer, just change the examples used in this book to use migrate instead of syncdb.
Example 4-2. Installing the Django database fixtures
$ cd /opt/graphite/webapp/graphite
$ sudo -E PYTHONPATH=/opt/graphite/webapp \
django-admin.py syncdb --settings=graphite.settings
/opt/graphite/webapp/graphite/settings.py:244: UserWarning: SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security
warn('SECRET_KEY is set to an unsafe default. This should be set in local_settings.py for better security')
Creating tables ...
...
... snipped for brevity ...
...
Creating table tagging_taggeditem
You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): admin
E-mail address: admin@mycompany.com
Password:
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)At this point, your web database is ready to use. Like any database, it’s a good idea to keep backups for disaster recovery. This process is out of the scope of this book, but it’s no different than backing up any other common database schema.
Configuring Carbon
Although Carbon uses up to eight different configuration files, only two of them are mandatory: carbon.conf and storage-schemas.conf. The former manages how many of each different Carbon daemons to run, which addresses and ports they listen on, and many of the settings that define their behavior, influence performance, and enable debugging. The latter defines the retention policy (precision and duration) for all Whisper metrics.
We’re going to look at an example of each that would be suitable for running on a small production Graphite server. You’re welcome to transcribe these settings verbatim, or you can copy the *.example versions Carbon includes at installation and make the changes I’ve recommended below.
Note that changes to the following files will only take effect after restarting the affected Carbon daemons.
carbon.conf
Most of Carbon’s configuration files use the INI file format, and carbon.conf is no exception. Each Carbon daemon type will have its own section starting with a bracketed header. If you’re running multiple instance of a certain type, each process will have its own section with the header specifying an alphanumeric identifier or index (e.g., [cache:1]; see Example 4-3).
Example 4-3. carbon.conf
[cache] USER = carbon MAX_CACHE_SIZE = inf MAX_CREATES_PER_MINUTE = 100 MAX_UPDATES_PER_SECOND = 1000 LINE_RECEIVER_INTERFACE = 0.0.0.0 PICKLE_RECEIVER_INTERFACE = 0.0.0.0 CACHE_QUERY_INTERFACE = 0.0.0.0 LOG_CACHE_HITS = False LOG_CACHE_QUEUE_SORTS = False LOG_UPDATES = False [cache:1] LINE_RECEIVER_PORT = 2003 PICKLE_RECEIVER_PORT = 2004 CACHE_QUERY_PORT = 7002
This configuration is much simpler than one you’re bound to use in production, but it also includes everything that should be important to help us get started. Graphite developers aim to use sane defaults where possible and avoid breakage (e.g., during upgrades) at all costs. In a similar vein, I’ve avoided listing a lot of the boilerplate where meanings and usage should be obvious; rather, I’d like to highlight a few key settings that should be relevant to any new user.
USER-
When this value is defined, the specified Carbon daemon will run as this user. Setting this to an unprivileged user is a good practice to adopt. However, it also means we’ll need to create the specified
carbonuser and group, as well as adjust some directory permissions (we’ll get around to this in the next section). Note thatUSERis unset by default, which makes me sad but is a reasonable default in helping users to get started when they don’t have a Graphite book at their disposal (wink). MAX_CACHE_SIZE-
This setting determines the amount of memory
carbon-cacheshould be allowed to use for caching. The default value (inf) assumes that your disk subsystem can keep up with the amount of creates and updates performed bycarbon-cache. If your write performance is poor, it’s a good idea to set this to a finite number representing the number of data points to store in memory (not the size in bytes). If the cache hits this limit, datapoints may be lost until the buffer catches up on writes. Each datapoint in Whisper is 12 bytes large, so you should be able to calculate some safe numbers based on your available memory. If you’re running Graphite in a heavily utilized virtual host, I strongly encourage you to setMAX_CACHE_SIZEto a non-infvalue. MAX_CREATES_PER_MINUTE-
Unsurprisingly, this limits the number of new Whisper files
carbon-cachewill create per minute. If this limit is reached in any one-minute window, any datapoints that arrive for a new metric will be dropped on the floor. That metric will be created the next time a datapoint arrives for that metric and this limit has not been reached. It can be tempting to set this value artificially high to guarantee new metrics, but you’ll end up shooting yourself in the foot since creates are significantly more expensive to process than updates. MAX_UPDATES_PER_SECOND-
This setting limits the number of Whisper updates performed by
carbon-cacheper second. Although it might seem confusing to measure creates per minute and updates per second, it makes sense when you consider that updates come in much more frequently than creates. Whencarbon-cachehits this limit, it responds by batching up multiple writes at once for efficiency. LINE_RECEIVER_INTERFACE-
The IPv4 or IPv6 network address Carbon listens on for plaintext metrics. Valid for all Carbon daemon types. Defaults to
0.0.0.0. PICKLE_RECEIVER_INTERFACE-
The IPv4 or IPv6 network address Carbon listens on for pickled metrics. Valid for all Carbon daemon types. Defaults to
0.0.0.0. CACHE_QUERY_INTERFACE-
The IPv4 or IPv6 network address the
carbon-cachedaemon listens on for cache queries (i.e., from the web application). Defaults to0.0.0.0. LOG_CACHE_HITS,LOG_CACHE_QUEUE_SORTS, andLOG_UPDATES-
These control various aspects of logging for
carbon-cacheand tend to be overly noisy. They can theoretically have an impact on the write volume of a Carbon host (particularly a very busy one) so I recommend disabling them to start.
Default Carbon Settings
If you’re ever unsure about a Carbon configuration setting, check the defaults in lib/carbon/conf.py. Using the source installation method we performed in this chapter, you’ll find this file at /opt/graphite/lib/carbon/conf.py. Open it in your favorite editor or pager utility (e.g., more or less) and look for the defaults object (in Python-ese, it’s referred to as a dict).
storage-schemas.conf
This file controls the retention policies for all of your metrics. It uses the same INI format as carbon.conf, although the section header names are arbitrary and used only to describe each schema (they must be unique). Every schema section needs to contain a pattern and one or more comma-delimited retentions.
Each pattern defines a regular expression to match on some portion (or the entire string) of a metric name. Carbon uses Python’s built-in regular expression syntax, which should be similar to anyone who’s used Perl-style expression syntax.
Retentions describe the precision and length of time to store each datapoint. Values can be represented as a raw integer or time description using the following units: seconds, minutes, hours, days, weeks, and years. The months descriptor is not supported since it contains an irregular number of days. The expressions in Example 4-4 are all equivalent:
Example 4-4. Equivalent expressions
10:604800 10s:7d 10seconds:1week
I’ve defined two different schemas in the following storage-schemas.conf (Example 4-5). The first one matches any metrics with the collectd. prefix (you might not have collectd running on your Graphite server, but that’s OK; it won’t hurt anything to include it). These metrics will be stored at two retention levels: 10-second precision for one week, and one-minute precision for one year. The former is considered the raw precision since we’re storing the original values intact. The latter is being downsampled into a rollup; as new metrics arrive, the rollup archive will continually be recalculated according to that metric’s aggregation method.
Example 4-5. storage-schemas.conf
[collectd] pattern = ^collectd\. retentions = 10s:1w, 60s:1y [default] pattern = .* retentions = 60s:1y
The second schema acts as a default schema for any metrics that didn’t already match a pattern defined before this one. Remember, the name “default” is completely arbitrary; it’s the pattern that determines whether it will match all metrics. The net result of our schema policy here is that all metrics will be stored at one-minute precision for one year, except for collectd-originating metrics, which will also be stored at ten-second precision for a week.
Tip
Settings in your storage-schemas.conf only affect Whisper files during creation. If you decide to change a retention policy after the corresponding metric file has already been created, you’ll either need to remove the affected Whisper file and let it re-create with the new policy or use the whisper-resize.py command to change the retention(s) manually.
storage-aggregation.conf
I hadn’t intended to cover this particular configuration file since it’s considered optional, but it really does play an important part if you’re intending to send any counter metrics or pre-aggregated metrics (e.g., using StatsD) to your Carbon service. You can consider this section a bonus, but please don’t tell my editor since I’ve asked to be paid by the word.
In the preceding section, I mentioned that the aggregation method has an effect in how we roll up metrics into lower-precision archives. Well, this is the file that allows you to specify the aggregation policy for your metrics. The syntax is very similar to storage-schemas.conf, with the addition of the xFilesFactor and aggregationMethod directives (Example 4-6). We already covered the purpose of each in the preceding chapter, so I urge you to jump back there if you need a quick refresher.
If you’re not already using an aggregator like StatsD that emits metrics like min, max, and count, you won’t need this file just to get started, but I think it merits a mention in this section anyway.
The same rules apply to storage-aggregation.conf regarding changes to an existing Whisper file; if you need to change the xFilesFactor or aggregationMethod, you’ll need to use whisper-resize.py to reconfigure the archives manually.
Example 4-6. storage-aggregation.conf
[min] pattern = \.min$ xFilesFactor = 0.1 aggregationMethod = min [max] pattern = \.max$ xFilesFactor = 0.1 aggregationMethod = max [sum] pattern = \.count$ xFilesFactor = 0 aggregationMethod = sum [default_average] pattern = .* xFilesFactor = 0.5 aggregationMethod = average
Some Final Preparations
If you followed my suggestion to run your Carbon daemons as an unprivileged USER, I applaud you. However, there are a few final housekeeping tasks we’ll need to perform before Carbon is ready to start (if you decided to run Carbon as a superuser, shame on you, but you can skip this section).
First, we’ll need to create the system user (Example 4-7).
Example 4-7. Creating the system user
$ sudo /sbin/groupadd carbon $ sudo /sbin/useradd -c "Carbon user" -g carbon -s /bin/false carbon
We also need to fix the ownership permissions for a number of storage and log directories. These allow the various Carbon daemons to write logs as the unprivileged user, create and update Whisper files, and in one case, guarantee the web server user (in this case, www-data; adjust as necessary based on your distribution) will be able to write to the SQLite web database:
$ sudo chmod 775 /opt/graphite/storage
$ sudo chown www-data:carbon /opt/graphite/storage
$ sudo chown -R carbon /opt/graphite/storage/whisper
$ sudo mkdir -p /opt/graphite/storage/log/carbon-{cache,relay,aggregator}
$ sudo chown -R carbon:carbon /opt/graphite/storage/log
Starting Your Carbon Daemons
You should now have at minimum a carbon.conf and storage-schemas.conf saved in your configuration directory (/opt/graphite/storage/conf, if you followed my instructions). You’ve also created a dedicated user to run your Carbon daemons under, and prepped your storage and log directories accordingly. We’re finally ready to start your Carbon service.
If you configured the single carbon-cache instance like we covered earlier, starting the process is as simple as:
$ sudo -E carbon-cache.py --instance=1 start
Many online examples use letters to distinguish between instances (a, b, c, etc.). However, I recommend using numbers to identify your daemon instances instead. This pattern is much easier to script or automate when you need to manage a sequence of processes. Say we’ve scaled up our number of cache processes to handle an expected spike in traffic. Starting them all at once would be as straightforward as:
$ for i in `seq 8`; do sudo -E carbon-cache.py --instance=${i} start; done
Starting carbon-cache (instance 1)
Starting carbon-cache (instance 2)
Starting carbon-cache (instance 3)
Starting carbon-cache (instance 4)
Starting carbon-cache (instance 5)
Starting carbon-cache (instance 6)
Starting carbon-cache (instance 7)
Starting carbon-cache (instance 8)
Stopping them is the same command, except using the stop action:
$ for i in `seq 8`; do sudo -E carbon-cache.py --instance=${i} stop; done
Sending kill signal to pid 20961
Sending kill signal to pid 20968
Sending kill signal to pid 20973
Sending kill signal to pid 20977
Sending kill signal to pid 20981
Sending kill signal to pid 20987
Sending kill signal to pid 20992
Sending kill signal to pid 20998
Configuring Graphite-Web
Setting up your Graphite web application is pretty much a walk in the park after dealing with the myriad configuration files and settings for the various Carbon daemons. There are only a few files we’ll need to prepare. If you’re still following along at home, we should be up and running in a few minutes.
local_settings.py
All of the Graphite’s web application settings are configured in the local_settings.py file, which in turn overrides the defaults in the settings.py file. Do not make changes to settings.py yourself, as this file gets overwritten during upgrades. We’ll get started by copying the sample file provided by Graphite-Web and then making our changes there:
$ cd /opt/graphite/webapp/graphite $ sudo cp local_settings.py.example local_settings.py
The following settings are really the only ones we need to set before we start using Graphite-Web. Truth be told, SECRET_KEY is the only mandatory edit, although there’s a very good chance you’ll want to set TIME_ZONE to avoid any possible confusion in your graphs. And instructing Graphite-Web to use memcached by setting it here with MEMCACHE_HOSTS is such an easy performance win (when you finally need it) that it would be silly not to do it now.
SECRET_KEY-
This value is used by Django for salting cryptographic hashes to improve application security. It should be set to a reasonably long and random value, such as the string output of
date | sha256sum. Note that if you happen to be running multiple web application servers behind a load balancer, this setting should be identical across servers. TIME_ZONE-
Controls the time zone for metric queries or rendered graphs. Because all data handling is performed server-side, this will affect virtually all aspects of the user experience. If you have users across multiple time zones, it’s a good idea to leave this at the default value (
UTC). MEMCACHE_HOSTS-
A list of memcached servers to be used for caching metric query results. For our purposes, we need to set this to
['127.0.0.1:11211'], where our memcached service is running by default.
After editing those values in local_settings.py, we just have one last task before we proceed with setting up the Apache configuration. The default LOG_DIR location is fine, but the permissions and ownership wouldn’t allow the Apache system user to write the necessary logfiles in place. We’ll just need to change ownership on the directory before we continue:
$ sudo chown www-data /opt/graphite/storage/log/webapp # Debian and Ubuntu $ sudo chown apache /opt/graphite/storage/log/webapp # RHEL, Fedora and CentOS
Setting Up Apache
As you may recall, we’re using Apache’s mod_wsgi to run Graphite as a WSGI application. We need a simple Python script dropped into place to let Apache know where to find all the relevant Graphite and Django files to load at runtime. Fortunately, the Graphite-Web project already includes a working example that we can copy without any changes:
$ cd /opt/graphite/conf $ sudo cp graphite.wsgi.example graphite.wsgi
Next we need to create a VirtualHost configuration for Apache. This example is intentionally simple, providing just the bare necessities to allow us to get Graphite running under the Apache web server. Note that the following access control directives may vary depending on the version of Apache your distribution offers.
This file should be saved or copied into place wherever your distribution prefers to store its Apache configuration files. For Debian or Ubuntu, this typically means saving it as /etc/apache2/sites-available/graphite.conf and enabling the configuration with a2ensite graphite. For RHEL, Fedora, or CentOS users, you’ll want to save it as /etc/httpd/conf.d/graphite.conf:
<VirtualHost *:80>
# wsgi script and permissions to read it
WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi
<Directory /opt/graphite/conf>
Require all granted
</Directory>
</VirtualHost>
There are quite a few other best practices I’d normally apply here, but we’ve kept it basic for the sake of expediency. Most administrators would be wise to enable SSL and some sort of basic authentication to keep out prying eyes.
Now we’re ready to start the web server:
$ sudo service apache2 start # Debian and Ubuntu $ sudo service httpd start # RHEL, Fedora and CentOS
You may prefer to run Graphite as a WSGI application under NGINX instead of Apache. Although we’re not going to cover that particular setup in this book, rest assured there are a number of good resources and examples online to help you get running in no time.
Verifying Your Graphite Installation
You cannot tell, but I am giving you a thumbs up.
N.E.P.T.R.
At this point, you should have a working Graphite installation with Carbon and Graphite-Web. Whether you got here the easy way using Synthesize and Vagrant, or by taking the deliberate route with a hands-on manual installation, I applaud you for taking the initiative to set up your own personal metrics collection and visualization engine. It’s about to get crazy fun all up in here—at least if you care about data and graphs like I do.
Carbon Statistics
Even though you’re not sending metrics to the Carbon listener yet, each of the Carbon daemons records its own individual metrics to Whisper. These are an easy way to verify that carbon-cache is doing something. But perhaps we should first just confirm that carbon-cache is running at all:
$ cat /opt/graphite/storage/carbon-cache-1.pid 11730 $ pgrep -fl carbon 11730 carbon-cache.py
OK, that looks good, but let’s do a quick spot check of the network ports carbon-cache should be listening on:
$ netstat -nltp | grep '[27]00' tcp 0 0 0.0.0.0:7002 0.0.0.0:* LISTEN 1236/python tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN 1236/python tcp 0 0 0.0.0.0:2004 0.0.0.0:* LISTEN 1236/python
Good, we can see carbon-cache has the plaintext listener on port 2003, the pickle listener on port 2004, and the cache query port on 7002. Now let’s check out the Whisper filesystem and make sure Carbon is writing its own statistics out to disk:
$ cd /opt/graphite/storage/whisper $ ls -l total 4 drwxr-xr-x 3 carbon carbon 4096 Jan 2 00:16 carbon $ find carbon carbon carbon/agents carbon/agents/synthesize-1 carbon/agents/synthesize-1/creates.wsp carbon/agents/synthesize-1/memUsage.wsp carbon/agents/synthesize-1/blacklistMatches.wsp carbon/agents/synthesize-1/whitelistRejects.wsp carbon/agents/synthesize-1/cache carbon/agents/synthesize-1/cache/bulk_queries.wsp carbon/agents/synthesize-1/cache/size.wsp carbon/agents/synthesize-1/cache/queues.wsp carbon/agents/synthesize-1/cache/queries.wsp carbon/agents/synthesize-1/cache/overflow.wsp carbon/agents/synthesize-1/cpuUsage.wsp carbon/agents/synthesize-1/updateOperations.wsp carbon/agents/synthesize-1/metricsReceived.wsp carbon/agents/synthesize-1/committedPoints.wsp carbon/agents/synthesize-1/avgUpdateTime.wsp carbon/agents/synthesize-1/errors.wsp carbon/agents/synthesize-1/pointsPerUpdate.wsp
Sure enough, carbon-cache is writing out all of its internal stats. Heck, we can even look at the last five datapoints for the cpuUsage on this carbon-cache:
$ whisper-fetch.py carbon/agents/synthesize-1/cpuUsage.wsp | tail -5 1422938640 0.795059 1422938700 0.793212 1422938760 0.843837 1422938820 0.799686 1422938880 0.783158
Feeding New Data to Carbon
That’s all Kool and the Gang, but what about creating our own bespoke metrics? Let’s fire a simple test datapoint off to the carbon-cache plaintext port and see what happens. If everything’s working as intended, we should have a new Whisper file with the datapoint recorded and available for retrieval:
$ echo "test.foo 1 `date +%s`" | nc localhost 2003 $ pwd /opt/graphite/storage/whisper $ ls -l total 8 drwxr-xr-x 3 carbon carbon 4096 Jan 2 00:16 carbon drwxr-xr-x 2 carbon carbon 4096 Feb 3 00:11 test $ ls -l test/ total 6160 -rw-r--r-- 1 carbon carbon 6307228 Feb 3 00:11 foo.wsp $ whisper-fetch.py test/foo.wsp | tail -5 1422940020 None 1422940080 None 1422940140 None 1422940200 None 1422940260 1.000000
Building Your First Graph
Are you anxious to start graphing some of these metrics yet? I’m really excited you’ve made it this far because this is where Graphite really turns the fun up to 11.0. Now that we’ve got some metrics in the system, let’s see what they look like dressed to the nines as line charts.
For what it’s worth, the graphs you’re about to see came from one of my Synthesize hosts, so you’re bound to see less (and different) metric history than mine. That’s OK—you know what they say: it’s not the size of your metrics history, it’s how you use it to support your incident response.
We’re going to walk through a few steps just to verify that Graphite-Web is working as intended. If you’re new to Graphite, this will also give you your first hands-on experience with the user interface.
Loading the web interface
Synthesize users who are using the Vagrant method should be able to load the Graphite window in your favorite browser at https://127.0.0.1:8443/. Other Synthesize users will need to connect to the HTTPS service at the IP address (or hostname) of the system where you ran the installation script. If you followed the instructions in this chapter, you should be able to load Graphite on the HTTP port of the intended system.
The first time you load the Graphite user interface, you’ll notice that it uses a somewhat conventional application layout: a navigation tree in the side panel to the left, a main panel containing the Composer window, and a title header at the top containing a series of links (e.g., Login, Documentation, Dashboard, etc.).
Most users should be familiar with the navigational icons and elements found here. Within the navigation tree, you’ll encounter nested folders such as Metrics that represent the structure of Whisper directories and files stored on the local filesystem. Not surprisingly, these also map to the hierarchy of dot-delimited metric names submitted to our Graphite server (or in Figure 4-1, emitted by carbon-cache).
Figure 4-1. Navigating metrics in the tree
The first time you click on any of these folders (until the next page reloads), the Graphite interface (i.e., client) makes an AJAX call to the backend Graphite web service, asking for any known objects (i.e., child folders or leaf elements, such as metrics). The service then returns a JSON object containing what it knows, which the client interprets and uses to redraw the tree view. If you’re familiar with the web inspector or developer view of modern browsers, you can even inspect the page yourself to watch the requests and responses; this is a great way to learn how Graphite-Web interacts with the underlying components.
Why do I mention all this gobbledygook? Because if this simple navigational example works, it’s your first hint that Graphite-Web, Carbon, and Whisper are all working in tandem. It means we have Whisper metrics stored on disk and that the web server’s system user has the necessary permissions to read them. High Five.
Adding metrics to a graph
Clicking on any metric node in the Metrics tree will add it to a graph embedded in the Composer window in the main panel. If the metric already exists in the graph (and hasn’t been modified with render functions), clicking on the metric node will remove it from the graph.
Tip
The Composer will try to keep you from adding the same metric unnecessarily to a graph. If you really need to render the same metric multiple times in a single graph (e.g., to compare to a timeshifted period), apply at least one render function to the first metric instance. This makes the metric definition unique, allowing you to add the same metric a second time by clicking on the node in the Metrics tree.
Here I’ve selected the metricsReceived metric listed under the carbon.agents.synthesize-1 namespace. A line series representing the available datapoints is rendered in the graph, and a simple legend appears at the bottom of the graph associating the metric with the color blue (Figure 4-2).
By also selecting the updateOperations metric, we can see in Figure 4-3 that our graph now contains two line series. The new metric has been associated with the color green, and both metrics are included in the legend.
You may have also noticed that the scale of our y-axis within the graph has changed substantially. Graphite is smart enough to automatically scale the axis so every metric is visually represented in the image. If it wasn’t, that large dip in updateOperations would’ve been cut off by the lower boundary of our graph.
Figure 4-2. Adding a metric to your graph
Figure 4-3. Adding another metric
Now is a great time to wind things up and regard our installation as a success. There’s a lot of exciting stuff ahead of us in the next chapter, so we’ll resume our graph party over there.
I truly hope you picked up a few useful tips in this chapter. We covered a lot of ground in terms of configuration settings and system commands, although this process was simplified a great deal to make it palatable to new users. We’ll cover far more advanced settings and configurations later on in this book, but it’s important that we’re able to lay a basic foundation and move forward with the interesting content for now.
Get Monitoring with Graphite now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
