Good organization is really the key to a good disaster recovery plan. If you have hundreds or thousands of backup volumes but can’t find them if you need them, what good are they? There is also the physical layout of the servers themselves. If they are all laid out in a standard way, recovering from a disaster is a whole lot simpler than if each server has its own unique layout.
Standardizing the layout of your servers is one of the more difficult things to do, since server configurations and OS configurations change over time. Look at the following list for some of the ways you can standardize, and standardize where you can. Experience has shown that it is worth the trouble to go back and restandardize. That is, it is worth the trouble to reimplement your new standard on your old servers.
This should be your standard everywhere. Keep your OS on one disk if possible. Recovering an OS that is spread out on multiple disks is very difficult. Also, keep the partitioning (or LVM partitioning) of all of your OS disks consistent. You don’t want to have to remember, “Oh yeah, this is the one with 1 MB of swap . . .”
If you have disks that serve the same purpose, partition them in the same way.
Decide on the best way to partition your database data disks, and partition all of them in the same way. For example, you might decide to fit as many 2 GB partitions as you can onto the disk. Anything left over can be used for those small databases that are always lurking around.
You need to keep track of your backup volumes. You need to be able to find any one of them at a drop of a hat. Here is a list of things you can do to ensure that:
Regardless of its name, each volume should have a unique volume serial number (volser #), which will identify that individual volume. Its name may change over time, but this number will always refer to that volume and that volume only.
If you have volumes in more than one location, you need a database. If you have people who use your backup volumes, you need a database. If you want to find your volumes ever again, you need a database. It can track a lot of information for you, including to whom you loaned a volume.
All tape media should be stored in such a way that the spindle, or axle, of the tape wheel, is horizontal—in the same way that a car’s axles are horizontal. Do not store tapes so that the axle of the tape reel is pointing upwards. This means that most tapes should be stored on their sides—not laying in a drawer somewhere. Tapes have been known to shift and lose their alignment if stored in that position for too long. (CD-ROM and optical media is less susceptible to this problem.)
The better the climate of your media storage area, the longer the media will last. If the area is just a normal office with unfiltered air and occasionally or even regularly rises to temperatures that feel warm to a human, your media is in the wrong place.
Media costs money. If you leave your backup volumes in an unlocked drawer, someone is liable to walk away with them. The cost of the media is not the problem, it’s the loss of data that is stored on them. Keep your media secured. Don’t let anyone but a select few have access to the media, and ensure that anyone else who is given access is logged. Remember, unless the data on the volume is encrypted, anyone with a backup drive can read it—no matter what file protections exist on your server.
Do an occasional inventory spot check of a random sample of volumes, perhaps once a month or quarter. Make sure that they are where you think they are. Then follow it up with a semiannual full inventory of all backup volumes.
A friend of mine used to say, “Online good, paper bad.” In the computer world, it is very good to have your documentation online. Online documentation is easier to update and easier to access during normal operations. However, it does have one drawback—it’s difficult to read in a disaster. With that in mind, you should put all your documentation eggs in one basket, and make that basket very easy to find.
Run a system layout program (such as the SysAudit
or SysInfo programs discussed in Chapter 4) on a regular basis and store the output in a
centralized location. For example, if you have automounter and a
central machine called admin, you
might store all SysAudit output in
You need to have well-documented procedures for how to do everything, from day-to-day system administration to how to rebuild your most important servers.
You also might want to consider having a special backup made of all your documentation. If you can fit such a backup on PC-style media (Zip, Jaz, or CD-ROM), it might make reading it in a disaster much easier, since many people on your IT staff may carry a laptop. A properly made CD-ROM can be read on either a Unix or Windows machine.
Put all of this documentation (from the system layout information to the actual procedures) in one place, so that you can create one tar backup of it. Whether this backup is to CD-ROM or to optical media or to a tape, it should be in one place to allow for easy retrieval.
You need to make sure that a copy of the executable needed to read your documentation is stored with that documentation. This definitely means backing up a copy of Word, Adobe Acrobat, or whatever document reader you use.