Backup & Recovery | O'Reilly Media

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I back up; therefore, I will be.

When I look at the title of this chapter, I think about the old Steve Martin stand-up routine in which he said that in philosophy class, “you learned just enough to screw you up for the rest of your life.” (Steve studied the important questions, like “Is it OK to yell ‘movie’ in a crowded fire house?” I promise not to do that.) However, “The Philosophy of Backup” did seem like an appropriate name for this chapter, since we’re going to talk about the why of backup. (We’ll also talk a little about the how, of course.)

A good backup and recovery system is essential for a company of any size. Unfortunately, IT doesn’t always get the budget it needs, and the backup system almost never gets the money that it needs. Well, if you agree that you need a very good backup system, but you don’t have enough money to pull that off, know that this book was written with you in mind. You need champagne backup on a beer budget. Welcome to the club.

Just because you have a small budget doesn’t mean you have to do without backup. Most of the backup systems in this book can be implemented in small environments for a few hundred dollars—including hardware.

Don’t worry, enterprise customers—there’s plenty in here for you as well. The more you use the techniques taught in this book, the more money you can save for other IT projects. By the time you’re done implementing all the ideas in this book, hopefully my next book will be done, which will be right up your alley. It will cover nothing but commercial data protection solutions, including multiplatform commercial backup and recovery systems, continuous data protection, near continuous data protection, data de-duplication backup systems, replication, and the like.

Now that you’ve read this far, you may find yourself asking questions like these:

Why should I read this book?
Can I really back up with open-source backup software?
Why should I be using disk?
Why should I back up at all?
How do I find a balanced way to back up (wax on/wax off)?

Let’s get started answering these questions.

If you’ve been doing system administration for some time, you may be asking yourself this question. There are many answers. Perhaps self-preservation is your primary motivator. You’d like to make sure you don’t lose your job the next time a disk drive dies. Perhaps you’ve already got a decent backup system and you’d just like to make it better. Maybe you are looking for some new ideas about how to deal with upcoming backup and recovery needs. What follows are some of the reasons

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A good backup and recovery system is essential for a company of any size. Unfortunately, IT doesn’t always get the budget it needs, and the backup system almost never gets the money that it needs. Well, if you agree that you need a very good backup system, but you don’t have enough money to pull that off, know that this book was written with you in mind. You need champagne backup on a beer budget. Welcome to the club.

Just because you have a small budget doesn’t mean you have to do without backup. Most of the backup systems in this book can be implemented in small environments for a few hundred dollars—including hardware.

Don’t worry, enterprise customers—there’s plenty in here for you as well. The more you use the techniques taught in this book, the more money you can save for other IT projects. By the time you’re done implementing all the ideas in this book, hopefully my next book will be done, which will be right up your alley. It will cover nothing but commercial data protection solutions, including multiplatform commercial backup and recovery systems, continuous data protection, near continuous data protection, data de-duplication backup systems, replication, and the like.

Now that you’ve read this far, you may find yourself asking questions like these:

Why should I read this book?
Can I really back up with open-source backup software?
Why should I be using disk?
Why should I back up at all?
How do I find a balanced way to back up (wax on/wax off)?

Let’s get started answering these questions.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

If you’ve been doing system administration for some time, you may be asking yourself this question. There are many answers. Perhaps self-preservation is your primary motivator. You’d like to make sure you don’t lose your job the next time a disk drive dies. Perhaps you’ve already got a decent backup system and you’d just like to make it better. Maybe you are looking for some new ideas about how to deal with upcoming backup and recovery needs. What follows are some of the reasons I think you should read this book.

Schadenfreude is a German word that means to take joy in the misfortunes of others. It’s why we watch those weird videos on the Internet where some idiot tries to do something stupid and ends up hurting himself. Each of the sidebars in this book is a true horror story that really happened to someone I know. These are not urban legends or horror stories passed on from admin to admin. These are firsthand encounters with disaster. There’s a schadenfreude element to reading these stories, of course. But each story also makes a point, and it was not just made up to make that point. The things that I warn about in this book really happen. This can be a very tough job if you are not prepared, so read closely. You might want to start by reading the sidebar “The One That Got Away” later in this chpater. It’s the story of the defining moment in my career.

“You mean to tell me that we have absolutely no backups of paris whatsoever?” I will never forget those words. I had been in charge of backups for only two months, and I just knew my career was over. We had moved an Oracle application from one server to another about six weeks earlier, and there was one crucial part of the move that I missed. I knew very little about database backups in those days, and I didn’t realize that I needed to shut down an Oracle database before backing it up. This was accomplished on the old server by a cron job that I never knew existed. I discovered all of this after a disk on the new server went south.

“Just give us the last full backup,” they said. I started looking through my logs. That’s when I started seeing the errors. “No problem,” I thought, “I’ll just use an older backup.” The older logs didn’t look any better. Frantic, I looked at log after log until I came to one that looked as if it were OK. It was just over six weeks old. When I went to grab that volume, I realized that we had a six-week rotation cycle, and we had overwritten that volume two days before.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I’ve heard it all. I’ve been accused of caring only about backups. It’s been said that I think the whole world revolves around a cartridge reel. I’ve said that someday the world’s going to crash, and I’m going to have the backup. The question is: how serious are you about protecting your data? To help you come to a decision on this matter, let’s talk about what happens if you don’t have good backups.

I was administering a QA group for a major software company. When the software did its setup, it created a directory in $HOME/foo so the software could install the user portion. A QA person was doing the install under root. The software created the directory $HOME/foo; literally, $HOME was the directory name. He submitted the bug fix, and decided to get rid of the useless directory. You’ve probably guessed it:

# rm -rf $HOME

(This was on a standard Unix system where $HOME for root was still /.)

Once I finally stopped laughing, I went and got the install media to rebuild the machine (no jumpstart image for that one, unfortunately, nor any backups for the various QA servers). Fortunately, most of the critical data was on the NFS server.

William Birch

To answer this question, you need to consider what kind of data you are backing up. This is a perfect time to include people who may not consider themselves computer people. Get input from other departments to answer this question. When all those 1s and 0s come together, just what kind of information are we talking about? Do you use manual accounting methods or are your company’s financial records stored in some accounting software somewhere? When a customer calls in and orders something, do you jot that down on a carbon-copied order form or do you enter it in some sort of order processing program? What about things like budgets, memoranda, inventories, and any other “paperwork” that you throw around from day to day? Do you keep copies of every important memo that you send, or do you depend on the computer for that?

If you’re like most people, you have grown quite dependent on these things we call computers. You forget how much of your work has been saved in the form of little magnetized bits spread out across a bunch of spinning platters. Maybe you work in an environment in which you’ve never lost a disk, so you’ve never had to do a restore. Maybe you’ve never fat-fingered a key and deleted an important file. If that’s the case, remember what my dad used to say: “motorcycle riders come in two types—those who have fallen and those who will fall.” The same is true of disk drives. If you’ve never had a failed disk drive, trust me, your turn is coming!

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Using a system that has no backups is like driving a car 100 miles an hour down a busy road the day after your insurance policy expires. Likewise, having a three-node, highly available cluster for a noncritical application is like having full coverage on your 20-year-old fifth car. Just as insurance plans have different levels of coverage and riders to cover various types of damage, different backup methodologies provide different levels of recoverability.

One memorable moment was when we had a 600 GB file server that hadn’t been properly backed up in a while. During a particularly hot weekend, both A/Cs handling the room failed, and temperatures soared. We shut everything down, waited for the A/Cs to be fixed, and started things back up after it cooled off a bit. Sure enough, two disks, physically next to each other in the same RAID4 array, failed. We were narrowly able to avoid total data loss by finding a spare disk and swapping control boards between it and one of the failed drives, which let it spin up and be accessed. We had the vendor courier us replacement drives the next day and then spent a lot of time fixing the backup server.

Theo Van Dinter

Not all environments need up-to-the-minute data recoverability. For many environments, recovering the systems up to last night’s backups is acceptable. For some environments, recovering the system even up to last week or month is OK. Spending thousands of dollars and hundreds of hours implementing the greatest backup solution in the world is a waste if you don’t need that level of coverage. This usually is not the problem for most sites; on the contrary, most sites don’t spend nearly enough money or effort on their backup and recovery systems. In other cases, however, money may be wasted on unnecessarily elaborate systems.

Recoverability requirements also vary from machine to machine within the same company. The amount of work that would be lost, or the possibility of adversely affecting a customer, may determine these requirements. For example, it may be considered acceptable for an employee or two to lose a day’s work spent on a few word processing documents. That is, unless it was the Senior Vice President’s assistant who was working on the departmental budget, in which case your mileage may vary

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Now that the philosophy lessons of Chapter 1 are over, it’s time to look at some of the important concepts behind backup and recovery, such as what to include, when to back it up, and more.

The casual reader might assume that this chapter is an introduction to basic backup concepts. While that is, in fact, the purpose of this chapter, it is also true that many seasoned administrators are unfamiliar with the ideas presented here. One reason for this is that administrators find themselves constantly being pulled away from “mundane” activities like backups for things that are thought to be more “important,” such as installing new servers and figuring out why systems are running slowly. Also, administrators may go several years without ever needing to perform a restore. The need to use your backups on a regular basis would undoubtedly change your ideas about their importance.

I wrote this book because backup and recovery has been my primary area of emphasis for several years, and I would like to share the lessons I’ve learned from this focused activity. This chapter provides an overview of how your backups should work. It also explains many basic yet extremely important concepts upon which any good backup plan should be based, and upon which all implementations discussed in this book are based.

Would anyone reading this book say that losing data is OK? I don’t believe so. Then why do we treat backups so lightly? Sometimes I feel like Rodney Dangerfield when I’m arguing for better backups—“I tell ya, I don’t get no respect, no respect.” Backups often aren’t considered during systems design. When a new server is purchased, does anyone ask for the impact on the current backup methodology? Some IT departments do not even have control over the purchase of new systems, because they are sometimes bought by other cost centers. Have you ever tried to explain to another department manager why his terabyte-sized database server isn’t going to get backed up to the standalone, gigabyte-sized tape drive that came with it?

Another often-overlooked issue is backup personnel. Have you ever tried to find the person in charge of backups? It’s often an extra duty that gets passed around, in a manner similar to the way my sister and brother and I argued over whose turn it was to wash the dishes. If you are lucky enough to have a dedicated person, it’s usually the most junior person in the company. I know, because that’s how I got my first job. In fact, that’s how many people get their first jobs. How can we give such low priority to something so important? Perhaps we should change that. Will one book change this long-standing hiring tradition? Probably not, but maybe it will help. At the very least, if the person in charge of backups has this book, that person has a complete guide to accomplishing the immense task that lies ahead.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The casual reader might assume that this chapter is an introduction to basic backup concepts. While that is, in fact, the purpose of this chapter, it is also true that many seasoned administrators are unfamiliar with the ideas presented here. One reason for this is that administrators find themselves constantly being pulled away from “mundane” activities like backups for things that are thought to be more “important,” such as installing new servers and figuring out why systems are running slowly. Also, administrators may go several years without ever needing to perform a restore. The need to use your backups on a regular basis would undoubtedly change your ideas about their importance.

I wrote this book because backup and recovery has been my primary area of emphasis for several years, and I would like to share the lessons I’ve learned from this focused activity. This chapter provides an overview of how your backups should work. It also explains many basic yet extremely important concepts upon which any good backup plan should be based, and upon which all implementations discussed in this book are based.

Would anyone reading this book say that losing data is OK? I don’t believe so. Then why do we treat backups so lightly? Sometimes I feel like Rodney Dangerfield when I’m arguing for better backups—“I tell ya, I don’t get no respect, no respect.” Backups often aren’t considered during systems design. When a new server is purchased, does anyone ask for the impact on the current backup methodology? Some IT departments do not even have control over the purchase of new systems, because they are sometimes bought by other cost centers. Have you ever tried to explain to another department manager why his terabyte-sized database server isn’t going to get backed up to the standalone, gigabyte-sized tape drive that came with it?

Another often-overlooked issue is backup personnel. Have you ever tried to find the person in charge of backups? It’s often an extra duty that gets passed around, in a manner similar to the way my sister and brother and I argued over whose turn it was to wash the dishes. If you are lucky enough to have a dedicated person, it’s usually the most junior person in the company. I know, because that’s how I got my first job. In fact, that’s how many people get their first jobs. How can we give such low priority to something so important? Perhaps we should change that. Will one book change this long-standing hiring tradition? Probably not, but maybe it will help. At the very least, if the person in charge of backups has this book, that person has a complete guide to accomplishing the immense task that lies ahead.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

If you can’t answer this question, there’s really no point in moving forward. The good thing is that it is a really easy question to answer. Just think about all of the various things that can happen to your data, and then look at all of the types of data that you have. You should be familiar with each business unit that creates data, and how that business unit would be affected if that data was lost or damaged. All of this becomes your business justification for moving forward.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Experience shows that one of the most common causes of data loss is that the lost data was never configured to be backed up. The decision of what to back up is an important one.

When trying to decide what files to include in your backups, take the most pessimistic technical person in your company out to lunch. In fact, get a few of them together. Ask them to come up with scenarios that you should protect against. Use these scenarios in deciding what should be included, and they will help you plan the “how” section as well. Ask your guests: “What are the absolute worst scenarios that could cause data loss?” Here are some possible answers:

An entire system catches fire and melts to the ground, leaving an unrecognizable mass of molten metal and blackened, smoking plastic.
Since this machine was so important, you had it replicated to another node right next to it. Of course, that machine catches fire right along with this one.
You have a centralized server that controls all backups and keeps a record of backup volume locations and what files are on what volumes, and so on. The server that blew up sits right next to this “backup server,” and the intense heat takes this system with it.
The disastrous chain reaction continues, taking out your DHCP and Active Directory servers, the NIS master server, the NFS and CIFS home directory servers, and the database server where you house the inventory of all your backup volumes with their respective locations. This computer also holds the telephone database listing all service agreements, vendor telephone numbers, and escalation procedures.
You haven’t memorized the number to your new off-site storage vendor yet, so it’s taped to the wall next to your backup server. You realize, of course, that the flames just burned that paper beyond recognition.
The flames set off the sprinkler system, and water pours all over your backup volumes. Man, are you having a bad day....

What do you do if one of these scenarios actually happens? Do you even know where to start? Do you know:

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This might appear to be the most straightforward topic. Everybody backs up their system every night, right? What’s the big deal? Actually, this could more aptly be titled “What levels do I run when?” It’s always a big question. How often do you run a full backup? How often do you run incremental backups? Do you run various levels of incrementals that back up just today’s changes or continuous incremental backups that back up everything since the last full backup? Everyone has her own answers to these questions. The only thing that is a definite is that there should be at least some level of backup every night. Before any further discussion on the topic, let’s define some terms.

The following are various backup levels. These terms are not used the same way by everyone.

Full/Level 0: A full backup.
Level 1: An incremental backup that backs up everything that has changed since the last level 0 backup. Repeated level 1 backups still back up everything since the last full/level 0 backup.
Levels 2–9: Each level backs up whatever has changed since the last backup of the next-lowest level. That is, a level 2 backs up everything that changed since a level 1, or since a level 0, if there is no level 1. With some products, repeated level 9 backups back up only things that have changed since the last level 9 backup, but this is far from universal.
Incremental: Usually, a backup backs up anything that has changed since the last backup of any type.
Differential: Most people refer to a differential as a backup that backs up everything that has changed since the last full backup, but this is not universal. In Windows, a differential is a backup that does not clear the archive bit. Therefore, if you run a full backup followed by several differential backups, they act like differential backups in the traditional sense. However, if you run even one incremental backup in Windows, it clears the archive bit, and the next differential backup backs up only those files that have changed since the last incremental backup. That’s why a differential backup is not synonymous with a level 1 backup.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Once you’ve decided when you’re going to back up, you have to decide how you are going to back up the data. But first, look at what types of problems you are protecting yourself from.

As stated earlier, how you want to do your restores determines how you want to do your backups. One of the questions that you must ask yourself is, “What are you going to protect yourself from?” Are the users in your environment all “power users” who use their computers intelligently and never make dumb mistakes? Would your company lose a lot of essential data if the files on your users’ PCs were accidentally deleted? If a hurricane took out your whole company, would it be able to continue doing business? Make sure that you are aware of all the potential causes for data loss, and then make sure your backup methods are prepared for all of them. The most exhaustive list of potential causes of data loss that I have seen is in another O’Reilly book called Practical Unix and Internet Security by Simson Garfinkel and Gene Spafford. Their list, with my comments attached, follows:

User error: This has been, by far, the cause of the biggest percentage of restores in every environment that I have seen. “Hey, I was sklocking my flambality file, and I accidentally pressed the jankle button. Can you restore it, please?” This one is pretty easy, right? What about the common question: “Can you restore it as of about an hour ago?” You can do this with continuous data protection systems and snapshots, but not if you’re running backups once a night.
System-staff error: This is less common than user error (unless your users have root or administrator privileges), but when it happens, oh boy, does it happen! What happens when you newfs your database’s raw device or delete a user’s document folder? These restores need to go really fast, because they’re your fault. As far as protecting yourself from this type of error, the same is true here as for user errors: either typical nightly backups or snapshots can protect you from this .

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

It doesn’t do any good to make really good backups only to have your backup volumes destroyed, lost, or misplaced. You need to have a well-defined process for storing your media.

If you’ve read this far, you know that I consider your backups very important. If your backups are important, isn’t the media on which they reside just as important? That goes without saying, right? Well, you’d never know it from most volume “libraries.” Volume “piles” is probably a more accurate term. How many computer rooms have you seen that have volumes spread out all over the place? They get stacked, piled, fall behind the systems, and a tape cartridge works really well as a coaster for a coffee mug. (We wouldn’t want to get any coffee rings on the new server, right?)

Have you ever really needed a volume and couldn’t find it? I’ve been there. It’s a horrible feeling to know that you’ve got the file on a volume, but can’t find the darn volume! Why, then, do we treat our backup volumes like so much dirty laundry? Organize your backup volumes! Label them, catalog them, give them unique names or numbers, and put them in some sort of logical order in some kind of storage container. Do it, or the backup demon will come to haunt you!

Your ability to perform a large recovery quickly is directly related to how well you organize your media.

What about that media cabinet that you’re using for your on-site volume storage? You don’t have one, you say? You’re using a file cabinet, you say? Well, use something, but if you can afford it, a number of companies make storage containers for media. They also make cabinets that can withstand fire. Spend the money; you’ll be glad you did. Doing a restore is so much less stressful when you can find the volume with no problem. Remember, though, that fireproof does not mean heat-proof. These types of media safes are meant to withstand brief fires that are quickly extinguished by a sprinkler system. If a fire burns for a long time right next to the container or raises the temperature in the room significantly, the volumes may be no good anyway. (This is another good reason why you also must store volumes off-site.)

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

I wish there were enough to say about this to make it a separate chapter, because it’s that important. I can’t tell you how many stories I have heard about people who waited until they needed a major restore before they tested their backups. That’s when they found out that they’d been using the wrong device or the wrong blocking factor, or that the device had I/O errors. This point cannot be stated strongly enough. If you don’t test your backups, you are guaranteed to get a surprise sooner or later.

It is important to test every type of restore. If you are testing filesystem backups, make sure you:

Restore many single files. Can you find the needle in the haystack?
Restore an older version of a file.
Restore an entire drive or filesystem, and compare your results with the original. Are they the same size, and so on?
Pretend that an entire system is down, and try to recreate it.
Pretend that a particular volume is bad, and force yourself to use an alternate backup.
Retrieve a few volumes from your off-site storage vendor.
Pretend that your backup server is destroyed, and try to recover from that. (This one’s tough!) This test is extremely important if you are using an open-source or commercial backup utility. Some products do not plan for this well, and you can find yourself in a real Catch-22 situation.

If you are testing database restores, make sure you:

Restore part of your database, pretending that you lost only one data file or disk drive, if this option is available.
Restore the entire database onto another server; this is where you learn about files that you are not including.
Restore the database up to a point in time, earlier than the present time (this is helpful practice for recovering from a DBA or user error).
Pretend that last night’s backup failed, and force yourself to use an older backup. Theoretically, if you have saved all your transaction logs to a backup volume, you should be able to use a backup that is weeks old and roll it forward to the present time using those logs. This is another strong argument for using transaction logs.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

If you are not monitoring your backups, they are not doing what you think they are doing—guaranteed. This is one pot that will not boil if you don’t watch it. Every backup should have a log that is examined daily. This can be automated as well. Here are some examples:

Give me a summary.

dump gives a whole bunch of messages that I couldn’t care less about, Pass I, Pass II, % done, and so on. When I’m monitoring the dump backups of hundreds of drives or filesystems, most of that is so much noise. What I really want to see is what got dumped, where it went, when it went, what level it was, and the ever-popular DUMP IS DONE message. To get a summary of just these lines, the first thing I do is use grep -v to exclude the phrases I don’t want, leaving only a few lines. This is much easier to review. This technique can also be applied to other Unix, Linux, and Mac OS backup commands.

Show me anything weird.

You can do this in either of two ways. If you know the phrases that show up when things go wrong, grep for those. Another way is to use grep -v to remove all lines you’re expecting and see what’s left. If there’s nothing, great! If there are lines left over, they are probably errors. You may see lines such as I/O error, Write error, or something else you don’t like to see in your backups.

If you want to apply this to a Windows task, you need to take advantage of some Unix emulators, such as Cygwin, UWIN, or GnuWin32, to allow you to run grep and other shell commands on a Windows system.

I don’t care how good your backups are; they can always be better. You could spend every waking hour tweaking and improving every piece of your backup program and know everything there is to know about backups, and they could still be better. My backups will never be good enough. There’s always a new bell or whistle on some other backup package, a bigger or smarter jukebox, a faster backup drive, or some scenario I thought of that I’m not covering. You must realize, however, that every change you make has a potential for data loss. A common thread that you will find in this book is that every time the human being enters into the equation, things can go wrong. You may be the best shell or Perl hacker in the world, and you will still make mistakes.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Don’t make a new change on your backup system and then roll it out to all your machines at once. Test it on a development system, or better yet, on a system that you don’t normally back up. That way you aren’t putting any backups in jeopardy. Another good practice is to test the change in parallel with what you’re already doing. The bigger the change, the more important it is to do a parallel conversion. This is especially true if you’re using a new method rather than just enhancing your current one. Don’t stop using your old method until you’re sure that the new one works! Follow a plan similar to this:

Test backup changes somewhere where they really won’t hurt anybody if they do something like, oh, crash the system!
Test the operation on a small scale on one system, using it in the same manner as you would in production. For example, if you are going to do both remote and local backups with this program, test both on a small scale.
Try to simulate every potential error the system might encounter:
- Eject a volume in the middle of the backup.
- Write-protect a volume.
- Reboot the system you are backing up while it is backing up.
- Drop the network connection, and power down a disk drive.
- Know the system and the errors for which it is testing, and simulate each one to test that section of your system.
Test on a small number of systems, preferably in parallel with your current method.
When you roll it out to all systems, definitely do so in parallel. One of the ways you can do this is to squeeze all your backups onto as few volumes as you can, then use the leftover drives to do the new backup in parallel. Your network guys might hate you, but it’s really the only way to do a true parallel conversion. When I converted to my first commercial backup utility, I ran in this mode for almost a year.
Only after you’ve tested and thoroughly documented your new system should you turn off the old method. Remember to keep documentation and programs around to restore data from the old system until all the old volumes have been recycled into the new system.
Also consider your older backup volumes. If you have volumes that are five years old, are you going to be able to read them on a new vendor’s backup solution? Will you even be able to read them in version 14 of your current software if your company began writing the archive volumes in version 2? Will the media itself even be readable?

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

We were going to call this section “Oh, and by the way,” but that seemed like a really weird heading.

One of the reasons that backups are unpopular is that people are worried that they might get fired if they do them wrong. People do get in trouble when restores don’t go right, but following the suggestions in this section will help you protect yourself from “recovery failure fallout.”

Self-preservation: Document, document, document

Have you ever tried to go on vacation? If you’re the only one who understands the restore process or the organization of your media, you can bet that you will be called if a big restore is required. Backups are one area of system administration in which inadequate documentation can really get you in trouble. It’s hard to go on vacation, get promoted, or do anything that would pull you away from the magical area that only you know. Your backups and restores should be documented to the point that any system administrator can follow them step by step in your absence. That is actually a good way to test your documentation: have someone else try to use it.

The opposite of good documentation is, of course, bad or nonexistent documentation. Bad documentation is the surest way to help you find a new job. If you do ever manage to take a real vacation in which you don’t carry a beeper, check your voice mail, or check your email—in short, watch out: Murphy’s Law governs vacations as well. You can guarantee that you, or more accurately, your coworkers, will have a major outage that week. If they crash and burn because you left them no guidelines for how to perform a restore, they will be looking for you when you return. You will not be a popular person, and you just might find yourself combing through the want ads.

On a number of occassions, a lack of documentation caused me to lose personal time. I remember one vacation during which I spent two to three hours on the phone every day. I remember spending long nights in computer rooms because no one knew which button to press next. But none of those memories is as strong as the time when my wonderful daughter, Nina, was born. Right about now, you’re probably saying “Aaah, that’s sweet.” It’s not what you think. Yes, she and my other daughter, Marissa, have given me a whole other reason to get up every day, but that’s not what this story’s about.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The chapters that follow explore in depth the various methods that you may employ to back up your systems, especially open-source tools. Most of these topics are also covered in documentation from the appropriate vendor or open-source team; this book is not meant to be a replacement for that documentation. Here, I try to explain things that are not covered in the documentation and possibly address some subjects more frankly than can a manual provided by the vendor.

Welcome to the world of backups.

BackupCentral.com has a wiki page for every chapter in this book. Read or contribute updated information about this chapter at http://www.backupcentral.com.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Basic backup utilities are the backup utilities upon which all noncommercial backup systems are built. They accomplish the important task of copying data from one place to another, and usually copying into another format (for example, tar). None of these tools have any built-in scheduling abilities, nor can they make a catalog to keep track of the backups that you make with them. If you want to perform these tasks, you’ll need some type of wrapper and scheduling application. This could be a simple batch script and a scheduled task on a Windows system, a shell script and cron entry on a Unix or Mac OS system, or one of the sophisticated open-source utilities covered later in this book.

Basic backup utilities include the native versions of dump, cpio, tar, and dd for Unix systems, ntbackup and System Restore for Windows systems, ditto for Mac OS systems, and the GNU versions of tar, cpio, and rsync that are available for all these platforms. Whether you’re just starting out in the backup world or you’re an experienced systems administrator, you need to be familiar with these utilities.

This chapter describes the benefits and pitfalls of several utilities. For all versions of Windows since NT, ntbackup is the only native choice for a traditional backup application, although you should also be familiar with System Restore. Mac OS X users running a version greater than 10.4 have a number of Unix-based backup tools available to them, including cpio, tar, rsync, and ditto. For commercial Unix systems, dump and restore are quite popular, but they’re not considered a viable option on Linux. dump is available on Mac OS, but it doesn’t support HFS+. After dump and restore, the native backup utility with the most features is cpio, but it is less user friendly than its cousin tar. tar is incredibly easy to use and is more portable than either dump or cpio. The GNU versions of tar and cpio have much more functionality than either of the native versions. If you have to back up raw devices or perform remote backups with tar or cpio, dd will be your new best friend. Finally,

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

This chapter describes the benefits and pitfalls of several utilities. For all versions of Windows since NT, ntbackup is the only native choice for a traditional backup application, although you should also be familiar with System Restore. Mac OS X users running a version greater than 10.4 have a number of Unix-based backup tools available to them, including cpio, tar, rsync, and ditto. For commercial Unix systems, dump and restore are quite popular, but they’re not considered a viable option on Linux. dump is available on Mac OS, but it doesn’t support HFS+. After dump and restore, the native backup utility with the most features is cpio, but it is less user friendly than its cousin tar. tar is incredibly easy to use and is more portable than either dump or cpio. The GNU versions of tar and cpio have much more functionality than either of the native versions. If you have to back up raw devices or perform remote backups with tar or cpio, dd will be your new best friend. Finally, rsync can be used to copy data between filesystems on Windows, Mac OS, Linux, and Unix.

This chapter begins with an overview of each of these backup utilities. It then goes into detail about the syntax for each command for both backup and recovery. Finally, near the end of the chapter, you’ll find an invaluable comparison chart that can be used as a quick-reference guide for comparing tar, cpio, and dump.

I went on one gig to fix a client’s “email” problem. Turns out it wasn’t an email problem; it was a DNS problem. They also asked me to look at their backups. What I found was appalling. They were doing backups by issuing commands to run dump out of cron:

They didn’t write a script; they just issued successive dump commands at different time intervals. Subsequent dumps were being executed before the previous one was finished.

They used the rewind device driver.

They were amazed they could fit everything on one tape!

Leon Towns-von Stauber (the author of Chapter 14) contributed this information about Mac OS backups.

What can make Mac OS X backups tricky is the default native filesystem format, HFS+, which is the advanced version of the legacy Macintosh Hierarchical File System. There are significant differences between HFS+ and the Unix File System (UFS), including support for

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

The ntbackup command activates the ntbackup GUI and, unlike with all other commands covered in this chapter, you cannot select what to back up with the ntbackup command itself. You have to select that from the GUI; however, you can run the GUI once, select what files to back up, and save that to a .bks file you specify on the command line later.

As with the other tools covered in this chapter, this section is not meant to replace the help page for ntbackup. It has many other options not covered here.

In addition to selecting which files are going to be backed up, you can also select values for a number of other options:

Type of backup (normal, copy, differential, or daily)
Type of target (disk or tape)
Name of target (for example, f:\backupfile.bkf )
Append or overwrite existing backups on target
Logging level (verbose, summary, or none)

These options can be specified as options on the command line or in the ntbackup GUI and saved as part of a .bks file. However, since you have to run the ntbackup GUI to create an ntbackup setup, we won’t cover the command-line switches in detail. Instead, we’ll show you how to get Windows to automatically create the command you need to run.

To create a simple backup with ntbackup, you need to create a backup options file using the ntbackup GUI, save it, then specify that options file when performing an ntbackup backup. Start the ntbackup GUI by typing ntbackup at the command prompt or by selecting Start→All Programs→Accessories→System Tools→Backup. From the Backup tab, select drives or directories to back up. Please note that you can back up the System State as well.

Next, you need to select various options about the backup. The two primary choices are the type of backup and where it will go. The available backup types are normal, copy, differential, and daily:

Normal (default): Back up the selected files and mark them as backed up.
Copy: Back up the selected files but do not mark them as backed up.
Incremental: Back up the selected files if they have changed since the last backup and do not mark them as backed up.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Anyone who has used Windows for a significant amount of time has had the experience of installing a new piece of software and having it render their Windows system useless. Previously, the only option would be to reinstall Windows and all your applications, but with System Restore this is no longer the case. If you’re able to boot into safe mode and select System Restore, you’ll probably be able to find a stable version of Windows to restore to. You’ll be back up and running in no time!

System Restore is a bit different from the other utilities in this chapter because it doesn’t create a backup in the traditional form, and you can’t use it as part of another tool. However, it’s a very important recovery tool that ships with Windows XP and later, and you should become familiar with it.

System Restore in Windows XP and later backs up the Windows registry and critical files to create a restore point. Windows automatically does this when it deems you are about to perform a significant event, such as the installation of a new driver or major patch. In addition, you can create your own restore points whenever you want, or at automated intervals using a scheduled task. You can then use any of the restore points that you or the system created to restore your system state to a previous point in time.

As mentioned previously, Windows actually creates a lot of restore points for you, assuming you haven’t disabled System Restore. To check whether System Restore is enabled, log in as a user in the Administrators group, and select Start→My Computer→Properties, and select the System Restore tab. You can then enable or disable it from this tab.

You must be logged in as Administrator or be in the Administrators group to use System Restore.

Anyone in the Administrators group can create a restore point at any time by selecting Start→All Programs→Accessories→System Tools→System Restore→“Create a restore point.” A dialogue box asks you to name the restore point you’re about to create. You can call it anything, such as Just before I Install Doom. The system then creates the restore point and gives it that name. You can then restore Windows to that point in time using System Restore.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

For many environments,


                  dump

may be all you need to ensure good-quality backups. There’s a lot of controversy surrounding dump, though, stemming from the fact that it doesn’t access the data through the filesystem the way most other backup utilities do. dump accesses the filesystem device directly. This is why it can back up files without changing their access times. However, it’s also why the manpages for dump have always said to unmount filesystems prior to backing them up. Of course, no one ever does that, hence the controversy.

Linux administrators should be aware that dump is not considered a good way to back up a Linux system, and dump doesn’t support the HFS+ filesystem in Mac OS. RedHat officially deprecated dump in RedHat 9, and the following quote from Linux Torvalds sums up the Linux community’s attitude towards dump:

“dump simply won’t work reliably at all even in 2.4.x: the buffer cache and the page cache (where all the actual data is) are not coherent. This is only going to get even worse in 2.5.x, when the directories are moved into the page cache as well. So anybody who depends on dump getting backups right [on a Linux system] is already playing Russian roulette with their backups. It’s not at all guaranteed to get the right results—you may end up having stale data in the buffer cache that ends up being “backed up”.... dump may work fine for you a thousand times. But it will fail under the right circumstances. And there is nothing you can do about it.”

You will have to make up your own mind on whether or not dump is right for you, but apparently dump isn’t the best way to back up a Linux system.

dump and restore are available on Mac OS, but they work only with the UFS filesystem. There is no hfsdump for the HFS+ filesystem, and I know of no plans to create such a tool.

To use dump and restore for regular system backups, you need to understand the following:

How to use dump to back up a filesystem (with the appropriate options)
How the backup ends up on the volume
How to get the table of contents of a

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

While writing this section, one phrase kept coming to mind, from a commercial for a motion-sickness medication in the U.S. called Dramamine. “The time to take Dramamine is too late to take Dramamine.” (By then, you’re already sick.) The same thing applies to learning how to use the restore utility. You need to become very familiar with the various ways in which you can use restore to retrieve data from a backup created with dump. If you are in the midst of a critical restore as you read this, don’t worry: this section is organized with that scenario in mind and includes every trick available in restore.

This next section assumes that you know the volume was made with dump and that you know its block size. If you do not have this information, see the section “How Do I Read This Volume?” in Chapter 23.

To make sure that you know the format and block size of a tape, try listing its table of contents. The following command produces the table of contents of a volume created with dump:

$ restore tbfy block_size device-name

For example, to read the table of contents of a dump tape (made with a blocking factor of 32) on /dev/rmt/0cbn, issue the following command:

$ restore tbfy 32 /dev/rmt/0cbn

If that works, then the rest is easy. (If not, read “How Do I Read This Volume?” in Chapter 23.)

Sometimes dump can write in a blocking factor that restore cannot read. This problem is usually very simple to get around. Once again, you need the block size in which the volume was written. Determine the volume’s block size as discussed in Chapter 23. Let’s assume that the block size of the volume is 65536. Use dd to read the volume, and pipe the output of dd to dump, giving “-” as the file argument. This tells restore to read its data from standard input.

# dd if=device-name bs=64k|restore tfy -

Why does this work? The blocking of data while writing to a volume drive actually changes how the data physically resides on the volume. The restore command needs to understand the blocking format to be able to read the volume. However, if you use dd to read the data from the volume, the data is put into a pipe. The

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

dump and restore have many capabilities. A good shell script can automate their use and can provide a very good safety net for that time when your disk goes south. However, these utilities do have their limitations:

There is no way with dump to get a consistent picture of an entire filesystem at any given moment in time.
The dump command is sometimes silent about open files and other problems, although it complains with a “bread error” if things get really confused.
When files are skipped, restore can actually make you think they are on the volume.
You do need to write scripts to work with dump, and scripts can have errors.
There are multiple versions of dump, not all of which play well with one another.
Like all native utilities, dump and tar lack online indexes like those available with commercial utilities. (Solaris’s version of dump does have an a option that performs some level of indexing, but it definitely isn’t the same as what you’d get with a commercial product.)

As long as you keep these issues in mind, you can get by for a long time using dump and restore and avoid spending anything extra for commercial software. Have fun!

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview

Table of Contents

Self-preservation: Document, document, document