Redis Cookbook

Chapter 4. Redis Administration and Maintenance

In this chapter, we’ll try to focus on recipes related to operating Redis servers, instead of programming applications or data modeling. These tasks vary widely, but include starting a Redis slave, upgrading an existing server, performing backups, sharding, and handling a dataset larger than your available memory.

Configuring Persistence

Problem

One of the advantages of Redis over other key/value stores like memcached is its support for persistence—in fact, it even comes preconfigured with this support. This functionality enables you to perform some operations that wouldn’t be possible otherwise, like upgrading your server without down time or performing backups.

Nevertheless, persistence should be configured in a way that suits your dataset and usage patterns.

Solution

The default persistence model is snapshotting, which consists of saving the entire database to disk in the RDB format (basically a compressed database dump). This can be done periodically at set times, or every time a configurable number of keys changes.

The alternative is using an Append Only File (AOF). This might be a better option if you have a large dataset or your data doesn’t change very frequently.

Discussion

Snapshotting

As previously stated, snapshotting is the default persistence mode for Redis. It asynchronously performs a full dump of your database to disk, overwriting the previous dump only if successful. Therefore, the latest dump should always be in your dbfilename location.

You can configure snapshotting using save seconds keys-changed statements in your configuration file, in the following format:

save seconds keys-changed

The snapshot will occur when both conditions match. A typical example that ensures that all your data is saved every few minutes is: save 600 1 which will perform a snapshot every 10 minutes if any key in your server has changed.

You can manually trigger snapshotting with the SAVE and BGSAVE commands. BGSAVE forks the main Redis process and saves the DB to disk in the background. Redis executes this operation itself if you have SAVE statements in your configuration file. SAVE performs the same operation as BGSAVE but does so in the foreground, thereby blocking your Redis server.

If you come to the conclusion that snapshotting is putting too much strain on your Redis servers you might want to consider using slaves for persistence (by commenting out all the save statements in your masters and enabling them only on the slaves), or using AOF instead. In particular, if you have a big dataset or a dataset that doesn’t change often, consider using AOF.

AOF

The Append Only File persistence mode keeps a log of the commands that change your dataset in a separate file. Like most writes on modern operating systems, any data logged to AOF is left in memory buffers and written to disk at intervals of a few seconds using the system’s fsync call. You can configure how often the AOF gets synched to disk by putting appendfsync statements in your configuration file. Valid options are always, everysec, and no.

Warning

Disabling fsync is not safe, as it leaves the decision to your operating system about when to actually write the data to disk.

AOF can be used together with snapshotting. But you might decide to suppress snapshots because they put too much load on your server. If you’re not snapshotting, be sure to write the AOF to a RAID array or have at least one Redis slave that you can recover data from in case of disaster.

Note

BGREWRITEAOF rewrites the AOF to match the current database. Depending on how often you update existing data, this will greatly reduce the size of the AOF. Since Redis 2.4, BGREWRITEAOF is run automatically. Nevertheless, should you wish to, you can also run it manually. The rewrite is done in the background, so as to not block your Redis server.

Starting a Redis Slave

Problem

Database slaves are useful for a number of reasons. You might need them to load-balance your queries, keep hot standby servers, perform maintenance operations, or simply inspect your data.

Solution

Redis supports master-slave replication natively: you can have multiple slaves per master and slaves connecting to slaves. You can configure replication on the configuration file before starting a server or by connecting to a running server and using the SLAVEOF command.

Discussion

In order to configure a Redis slave using the configuration file, you should add the following to your redis.conf:

slaveof master-ip-or-hostname masterport

Start or restart the server afterwards. Should your Redis master have password authentication enabled, you’ll need to specify it as well:

masterauth master-password

If you want to turn a running Redis server into a slave (or switch to a different master), you can do it using the SLAVEOF command:

SLAVEOF master-ip-or-hostname [masterport]

As in the previous example, if you’re using authentication, you’ll need to specify it beforehand:

CONFIG SET masterauth password

Keep in mind that should your server restart, this configuration will be lost. Therefore, you should also commit your changes to the configuration file.

Note

CONFIG SET allows you to read configuration parameters from a running Redis server. CONFIG GET enables you to set configuration parameters on a running Redis server. Please refer to the documentation for these commands’ parameters.

Using Redis As a Memcache Replacement

Problem

Memcache is a widely used, high-performance distributed caching system, providing a simple, lightweight, in-memory key value store. Given that Redis supports all of Memcache’s features, you can in fact use Redis instead of Memcache in your caching infrastructure and at the same time take advantage of some of the extra features available in Redis.

Solution

Our solution for this problem focuses on configuring Redis to match two of the major differences it has from memcache: persistence and maximum memory usage. Because Memcache doesn’t support any kind of persistence, we’ll disable it in Redis, and configure the maximum amount of memory used by Redis (a mandatory setting in memcache).

Discussion

In a previous chapter we explained how to configure the different persistence modes in Redis, but we didn’t explain how to disable them. The answer is simple, though: just remove all the save statements from your redis.conf and set appendfsync to no. Keep in mind that if you haven’t configured persistence, it’s probably enabled, because it’s the default setting.

When it comes to maximum memory usage, Redis allows you to configure both the amount of memory and the eviction policy used when the limit is reached. The amount is quite straightforward: adding maxmemory 1g to your configuration specifies a maximum memory usage of 1 gigabyte. The maxmemory-policy setting has six different policies, which determine what action is taken once the memory limit is hit:

volatile-lru

Removes a volatile key (one that has expiration set) using an LRU (least recently used) algorithm.

allkeys-lru

Removes a key from the database using an LRU algorithm.

volatile-random

Removes a volatile key at random from the database.

allkeys-random

Removes a random key from the database.

volatile-ttl

Removes the key with the lowest TTL (closer to expiration).

noeviction

Disables eviction. Write operations to the database will raise an error if they’d exceed the maxmemory setting.

As with any other Redis configuration variable, these can be set programatically using CONFIG SET.

Upgrading Redis

Problem

At some point in the life of your system you might need to upgrade Redis. Unfortunately, Redis can’t do online binary upgrades, and doing a server restart means that your application won’t be able to talk to Redis for a (possibly long) period of time. But that doesn’t mean that there aren’t other ways to achieve it without incurring downtime. You might also want to move your current Redis database to another system for maintenance purposes, a hardware upgrade, etc.

Solution

Our solution will involve starting a new Redis server in slave mode, switching over the clients to the slave and promoting the new server to the master role. To make the example easier to understand, let’s assume we have a Redis server listening on port 6379.

Note

It might be easier to start the slave on a new server than on the existing one. This is because of memory requirements, and because you can reuse the same configuration file, directories, and port for the slave, changing only the hostname or IP address.

1. Install the new Redis version without restarting your existing server.

2. Create a new redis.conf, specifying that Redis runs on port 6380 (assuming you’re on the same system—if you’re not, you can still use 6379 or any other available port) and a different DB directory (you don’t want to have 2 Redis servers reading or writing the same files).

3. Start the new server.

4. Connect to the new server and issue the command:

   SLAVEOF localhost 6379

   This will trigger a BGSAVE on the master server, and upon completion the new (slave) server will start replicating. You can check the current status using the INFO command on the slave. When you see master_link_status:up, the replication is active.

5. Since your new Redis server is now up-to-date, you can start moving over your clients to this new server. You can verify the number of clients connected to a server with the INFO command; check the connected_clients variable.

6. When all your clients are connected to the slave server, you still have two tasks to complete: disable the replication and shut down the master server.

Note

INFO returns information about the server including replication status, uptime, memory usage, number of keys per database and other statistics.

1. Connect to the slave server and issue:

   SLAVEOF NO ONE

   This will stop replication and effectively promote your slave to a master. This is important, because master servers are responsible for sending expirations to their slaves.

2. Now connect to your old master server and issue:

   SHUTDOWN

   The old master server will perform a SAVE and shutdown.

3. Your new Redis system is up and running, but make sure that all your configuration files, init scripts, backups, etc., are pointing to the right location and starting the correct server. It’s easy to forget those routine operations, but you should, at the very least, certify that nothing wrong will happen in case of a server restart.

Discussion

Doing an online upgrade has a couple of (possibly steep) requirements: you need to able to point your Redis clients to another server, either by use of a proxy, by having failover built-in to your clients (so that they connect to a different server once you bring the master down), or just by simply tell them to connect to another server. You’ll also need to have at least twice as much memory available (possibly in a different system).

Beware that doing this might be dangerous, depending on how different the Redis versions you are upgrading from and to. At the very least, it should be safe for updates of minor versions of Redis. For major upgrades, each has caveats. Like every other maintenance operation, make sure to test before doing it on your production servers.

Backing Up Redis

Problem

One issue comes up frequently when talking about NoSQL databases is backing up your data. The notion that these are hard to back up, however, is mostly a misperception since most of the techniques that you’d use to backup a relational database can also be used for NoSQL databases.

If, for some distributed databases, grabbing a point-in-time snapshot of your data might be tricky, this is certainly not the case with Redis. In this section, we’ll explain how to achieve it depending on which Redis persistence model you’re using. We’ll assume you’re running your servers on Linux, although filesystem-specific functionality might also be available for other platforms.

Solution

Our proposed solution is heavily dependent on your Redis persistence model:

• With the default persistence model (snapshotting), you’re best off using a snapshot as a backup.

• If you’re using only AOF, you’ll have to back up your log in order to be able to replay it on startup.

It’s up to you to store your backup properly. Ideally, you’ll store at least a couple of copies of it, have at least one offsite, and do it in a fully automated way. We’ll try to explain how to do backups for the different persistance models, but be sure to test your own procedures. Be sure to also test your recovery procedures regularly.

Keep in mind that backing up your data might increase the strain on your production systems. It’s probably a good idea to perform the backups on a slave Redis instance, and to actually have slaves running at all times because promoting a new server to master is probably quicker than restoring a backup.

Discussion

Snapshotting

Snapshotting is the default Redis persistance model. As mentioned earlier, depending on your settings, Redis will persist its data to disk if m keys changed in n seconds. When using this persistence mode, performing a backup is really simple. All you have to do is copy the current snapshot to another location.

Warning

Use a copy, not a move, because if Redis crashes and restarts and the snapshot is not there, you will end up losing all your data! However, keep in mind that you should disable snapshotting while you’re copying the file; otherwise, you could end up with a corrupted backup.

In case you want an up-to-date snapshot (instead of using the last one Redis did according to your settings) you can trigger it by issuing:

redis-cli BGSAVE

and then waiting for the dump file to be updated. Be sure to compress the snapshot before backing it up. That will probably reduce its size by at least a factor of 10.

Restoring a snapshot file is also quite simple. Simply shut down the server, put the snapshot you want to restore in the dbfilename location configured by redis.conf, and then start the server. This order is important, because when Redis shuts down, it performs a snapshot, thus overwriting this file.

Append-Only Log (AOF)

If you’re using the AOF as the only persistence mode (you can also use it together with snapshotting) the easiest way to do a backup is still to perform use a snapshot as described in the previous section. However, if you’re using AOF, you’re most likely worried about losing data between snapshots. You may also be avoiding snapshots because they put too much load on your server.

In order to recover when using the AOF, just do the same procedure you would for snapshotting, but instead put your backup in the AOF location. On startup, Redis will simply replay the log.

Be sure to remember to run BGREWRITEAOF regularly if you’re using AOF.

Should your Redis server refuse to start due to a corrupted AOF—which can happen if the server crashes or is killed while writing to the file—you can use the redis-check-aof utility to fix your AOF:

redis-check-aof --fix filename

High Availability with Sentinel

Problem

As with any other distributed system, when you develop applications or services using Redis, you’re introducing a new possible point of failure. Given that—until the release of Redis Cluster—you’re probably relying on a single master instance, so a failure in that node or server could cause downtime or a service disruption.

Solution

Sentinel was released with Redis 2.6. Redis Sentinel is a distributed monitoring system that continuously checks the state of your Redis instances and performs automatic failover, allowing Sentinel-aware clients to reconnect to the new master. This makes your applications more resilient to failures, because they can recover quickly if you have at least one running slave that can be promoted to master.

Discussion

In order to use Sentinel, you need to use a configuration file:

sentinel monitor master-name ip redis-port quorum
        sentinel down-after-milliseconds master-name milliseconds
        sentinel failover-timeout master-name milliseconds
        sentinel parallel-syncs master-name numslaves

Start Sentinel with:

redis-server /path/to/sentinel.conf --sentinel

Sentinel will fetch the running slaves from your master and start monitoring your cluster.

Keep in mind that the quorum is the minimum amount of Sentinel nodes that need to agree in order for a Redis master server to be considered down and trigger a failover. You should therefore have at least the same number of Sentinel nodes as specified in your quorum, preferably more.

You can also define multiple masters for your Sentinels to watch. For more in-depth information, please refer to the sample Sentinel configuration file.

In order for your application to be Sentinel-aware and be able to continue operating in a failover scenario, your clients must communicate directly with the Sentinel nodes using the Redis protocol. This is made easier by using a Sentinel-aware client, such as redis-sentinel, which will reconnect to the new master in the event of a failover.

You can use redis-sentinel in the exact same way you’d use a regular Redis client, except that instead of providing a host and port for your Redis master, you provide a master-name and a list of Sentinel nodes:

        require 'redis-sentinel'

        $r = Redis.new(master_name: "master-name",
                      sentinels: [{host: "first-sentinel-host", port: 26379},
                                  {host: "second-sentinel-host", port: 26380}])

Sharding Redis

Problem

Sharding is a horizontal partitioning tecnique often used with databases. It allows you to scale them by distributing your data across several database instances. Not only does this allow you to have a bigger dataset, as you can use more memory, it will also help if CPU usage is the problem, since you can distribute your instances through different servers (or servers with multiple CPUs).

In Redis’s case, sharding can be easily implemented in the client library or application.

Solution

Because Redis Cluster is still under development and should only be released sometime in late 2014, sharding is a useful tecnique for scaling your application when your data no longer fits in a single server.

Currently there are three possibilities when it comes to sharding Redis databases:

Use a client with built-in sharding support

At this point, most Redis clients don’t support sharding. Notable exceptions are:

redis-rb, a Ruby client

Predis, a PHP client

Redisent, a PHP client

Rediska, a PHP client

Jedis, a Java client

scala-redis, a Scala client

redis-shard, a Node.js library

Build sharding support yourself on top of an existing client

This involves some programming that might not be too hard if you understand your dataset and applications thoroughly. At the very least, you’ll have to implement a partitioning rule and handle the connections to the different servers.

Use a proxy that speaks the Redis protocol and does the sharding for you

Redis Sharding is a multiplexed proxy that provides sharding to any Redis client. Instead of connecting directly to your Redis servers, you start a proxy and connect to it instead. Unfortunately at this moment, sharding doesn’t support resharding on the fly, so you’ll be unable to change the configuration of the cluster with the proxy running.

Discussion

If you decide to implement sharding yourself, you should probably use consistent hashing. This will ensure a minimal amount of remapping if you add or remove shards.

Sharding doesn’t remove the need for replication. Make sure your cluster is redundant so that the loss of a server doesn’t imply any loss of data. Jeremy Zawodny described on his blog the setup used at Craiglist, and Salvatore has written on the subject as well.

Something else to keep in mind is that (depending on your implementation) you will not be able to perform some operations that affect multiple keys, because those keys might be in different shards (servers). If you rely on these operations, you’ll need to adjust your hashing algorithm to ensure that all the required keys will always be in the same shard.