Unix Backup and Recovery

Step 3: Organize Everything

Good organization is really the key to a good disaster recovery plan. If you have hundreds or thousands of backup volumes but can’t find them if you need them, what good are they? There is also the physical layout of the servers themselves. If they are all laid out in a standard way, recovering from a disaster is a whole lot simpler than if each server has its own unique layout.

Standardized Server/Disk Layout

Standardizing the layout of your servers is one of the more difficult things to do, since server configurations and OS configurations change over time. Look at the following list for some of the ways you can standardize, and standardize where you can. Experience has shown that it is worth the trouble to go back and restandardize. That is, it is worth the trouble to reimplement your new standard on your old servers.

The root disk

This should be your standard everywhere. Keep your OS on one disk if possible. Recovering an OS that is spread out on multiple disks is very difficult. Also, keep the partitioning (or LVM partitioning) of all of your OS disks consistent. You don’t want to have to remember, “Oh yeah, this is the one with 1 MB of swap . . .”

Same-size disks

Partition all of your same-size disks exactly the same way, if possible. Consistency makes swapping them in and out very easy and gives you a lot of flexibility.

Same-function disks

If you have disks that serve the same purpose, partition them in the same way.

Database data disk

Decide on the best way to partition your database data disks, and partition all of them in the same way. For example, you might decide to fit as many 2 GB partitions as you can onto the disk. Anything left over can be used for those small databases that are always lurking around.

Application disk

Usually, the best thing to do here is make it one big disk, while reserving that first cylinder again. (It’s a good habit to get into.)

Media Organization

You need to keep track of your backup volumes. You need to be able to find any one of them at a drop of a hat. Here is a list of things you can do to ensure that:

Unique alphanumeric volser#

Regardless of its name, each volume should have a unique volume serial number (volser #), which will identify that individual volume. Its name may change over time, but this number will always refer to that volume and that volume only.

Database to track volser#, name, type, date used, location, “loaned to”

If you have volumes in more than one location, you need a database. If you have people who use your backup volumes, you need a database. If you want to find your volumes ever again, you need a database. It can track a lot of information for you, including to whom you loaned a volume.

Bar code system

Bar codes are useful for more than tape libraries. You can purchase a bar code scanner rather inexpensively and use it to track the movement of your volumes.

Proper media storage

All tape media should be stored in such a way that the spindle, or axle, of the tape wheel, is horizontal—in the same way that a car’s axles are horizontal. Do not store tapes so that the axle of the tape reel is pointing upwards. This means that most tapes should be stored on their sides—not laying in a drawer somewhere. Tapes have been known to shift and lose their alignment if stored in that position for too long. (CD-ROM and optical media is less susceptible to this problem.)

Temperature and humidity

The better the climate of your media storage area, the longer the media will last. If the area is just a normal office with unfiltered air and occasionally or even regularly rises to temperatures that feel warm to a human, your media is in the wrong place.

Physical security

Media costs money. If you leave your backup volumes in an unlocked drawer, someone is liable to walk away with them. The cost of the media is not the problem, it’s the loss of data that is stored on them. Keep your media secured. Don’t let anyone but a select few have access to the media, and ensure that anyone else who is given access is logged. Remember, unless the data on the volume is encrypted, anyone with a backup drive can read it—no matter what file protections exist on your server.

Spot checks and full inventories

Do an occasional inventory spot check of a random sample of volumes, perhaps once a month or quarter. Make sure that they are where you think they are. Then follow it up with a semiannual full inventory of all backup volumes.

For a detailed example of the application of all of the above media organization concepts, see Section 2.8.2.1 in Chapter 2.

Put Electronic Documentation in One Place

A friend of mine used to say, “Online good, paper bad.” In the computer world, it is very good to have your documentation online. Online documentation is easier to update and easier to access during normal operations. However, it does have one drawback—it’s difficult to read in a disaster. With that in mind, you should put all your documentation eggs in one basket, and make that basket very easy to find.

Output from a system layout program

Run a system layout program (such as the SysAudit or SysInfo programs discussed in Chapter 4) on a regular basis and store the output in a centralized location. For example, if you have automounter and a central machine called admin, you might store all SysAudit output in /net/admin/client_name/SysAudit.out.

Procedures

You need to have well-documented procedures for how to do everything, from day-to-day system administration to how to rebuild your most important servers.

Avoid Those Catch-22 Situations

Planning for a disaster is difficult to do. You have to keep in mind the catch-22 situations that can surprise you. I remember when one of them happened to me. We were quite proud of our media inventory system (see Section 2.8.2.1 in Chapter 2). The database was well defined and constantly updated. We could find any volume at any time—as long as the database was available. What do you suppose we had to do when the system that contained the database went down? It wasn’t easy, I tell you, to find that volume. Luckily, we had the volume name and its bar code number on the volume itself. Once our backup software told us which volume it wanted, we simply searched high and low until we found it. After this little scenario, we changed the way our volumes were inventoried. We found out that the off-site storage company had a customer-defined field that we weren’t using. All we had to do was feed them the names of the volumes associated with each bar code. That way, the next time we needed a volume and did not have the database, we could ask them for it.

Files on Zip/Jaz/CD-ROM

You also might want to consider having a special backup made of all your documentation. If you can fit such a backup on PC-style media (Zip, Jaz, or CD-ROM), it might make reading it in a disaster much easier, since many people on your IT staff may carry a laptop. A properly made CD-ROM can be read on either a Unix or Windows machine.

One tar volume

Put all of this documentation (from the system layout information to the actual procedures) in one place, so that you can create one tar backup of it. Whether this backup is to CD-ROM or to optical media or to a tape, it should be in one place to allow for easy retrieval.

Make sure that the reader (Word, Adobe Acrobat, browser) is on the volume

You need to make sure that a copy of the executable needed to read your documentation is stored with that documentation. This definitely means backing up a copy of Word, Adobe Acrobat, or whatever document reader you use.