Preface
I hope you learn half as much reading this book as I did writing it. This was quite an interesting project, where we took the original book and expanded its scope so much that we had to change its title. I wrote Unix Backup and Recovery seven years ago, and a lot has changed since then—both in the industry and in my life. The biggest change in the industry has been the proliferation of Windows, Mac OS, Exchange, and SQL Server in the data center. (I never saw the Apple Xserve coming.)
The biggest change for me has been having my eyes opened to backup and recovery applications beyond those considered “traditional.” It’s true that I spend most of my professional life consulting with large companies that spend enough on backup software and hardware to fund a small army. I enjoy doing that. It’s very rewarding to show a company how to save millions of dollars a year and make their backups and restores faster and more reliable in the process. (By the way, if you need help with your backup system, drop an email to curtis@backupcentral.com—that’s what I do for a living.)
I also spend a good deal of the time traveling the world speaking to users about how to do this themselves. And when I do, I always get questions like these:
I got a quote for backup software from XYX, and they want $XXXX for backup software! Where am I supposed to get that kind of money!?
I couldn’t afford backup software from XYZ, so we bought ABC instead, and it stinks. Can you recommend something better?
None of the commercial utilities can back up my MySQL or PostgreSQL database. How do I do that?
How do I do bare-metal recovery on ABC operating system?
Aren’t there open-source utilities that do this kind of thing?
So while I’m actually preparing to write my next book on how to select, install, and manage commercial backup software systems, I felt that this book needed to come first. This book is aimed at the people who feel that the commercial software products aren’t meeting all their needs.
Perhaps you’re a small shop that can’t spend $10,000 just to get decent backup software. Perhaps you’re already using a commercial backup software package, but you don’t want to spend thousands of dollars on their agent to back up your DB2 databases, or you can’t find anybody to back up your MySQL or PostgreSQL databases. This book is about giving you options—free options.
Almost everything I talk about in this book is either included with your operating system or application, or is available as an open-source project. (The commercial products I do mention cost only $99.) You may be amazed at what you can do for free or almost free.
I Wish I’d Had This Book
I wanted to write a book that would ensure that no one would ever have to start from scratch again, and I believe that my contributors and I have done just that. It contains every backup tool that I wish I had when I first entered the backup business and every lesson and trick that I’ve learned along the way. It covers how to back up and recover everything from a basic Linux, Windows, or Mac OS workstation to a complicated DB2, Oracle, or Sybase database—and a lot of things in between. Whether your budget barely stretches to cover the cost of the backup media or allows you to buy a silo bigger than your house, this book has something for you. Whether your task is to figure out how to back up, with no commercial utilities, an environment such as the one I first encountered or to choose from among more than 50 commercial backup utilities, this book will tell you how to do it. With that in mind, let me mention a few things about this book that are unique.
Only the Recovery Matters
As my friend Joe Fitzpatrick used to tell me, “No one cares if you can back up—only if you can recover.” Yet how many backup chapters have you read that dedicate less than 10 percent to recovery? You won’t find that in this book. I have tried very hard to ensure that recovery is given equal treatment.
Products Change
Some people may be surprised that there are no product names mentioned in the commercial backup section. I did this for several reasons, the main one being that products change constantly. It would be impossible to keep this book up to date with more than 50 backup products that are available for Unix alone. In fact, the book would be out of date by the time it hit the shelves. Instead, this book explains the concepts of commercial backup and recovery software, allowing you to apply those concepts to the claims that the vendors are currently making. Up-to-date information about specific products is available on http://www.backupcentral.com.
Backing Up Databases Is Not That Hard
If you’re a database administrator (DBA), you may not be familiar with the commands necessary to back up your database. If you’re a system administrator (SA), you may not be familiar with the architecture of the database platform your DBA is using. Both concepts are explained in detail in this book. I explain the backup utilities in plain language so that any DBA can understand them, and I explain database architecture in such a way that an SA, even one who has never before seen a database, can understand it.
Bare-Metal Recovery Is Not That Hard
One of these days you will lose the operating system disk for an important system, and you will need to recover it. This is called a bare-metal recovery. The standard recovery method described in many backup products’ documentation is to install a minimal operating system and restore on top of it. This is the worst possible method to do a bare-metal recovery of a system; among other problems, you end up overwriting some of the system files while the system is running from the very disk to which you are trying to restore. The best ways to do bare-metal recoveries for AIX, Solaris, HP-UX, Windows, Linux, and Mac OS are covered in detail in this book.
How This Book Is Organized
This book is divided into six parts, which are described in the following sections.
Part I
Part I of this book contains just enough information to whet your backup and recovery appetite.
- Chapter 1, The Philosophy of Backup
Describes my philosophy about backup, such as why you should back up, and a little bit about how to do it, too.
- Chapter 2, Backing It All Up
Goes into detail about the essential elements of a good backup and recovery system.
Part II
This section covers the basic backup utilities that are available to back up your system, and several open-source backup systems to help you manage those backups.
- Chapter 3, Basic Backup and Recovery Utilities
Covers the basic backup and recovery utilities you’re likely to find in Unix, Windows, or Mac OS, including
dump
,tar
,cpio
,dd
,ditto
,ntbackup
, andrsync
.- Chapter 4, Amanda
Covers the ever-popular Advanced Maryland Disk Archiver, or Amanda.
- Chapter 5, BackupPC
Explains the disk-only backup system called BackupPC, which can actually back up far more than just your PC.
- Chapter 6, Bacula
Covers Bacula. It roams the data center at night and sucks the vital essence from your computers.
- Chapter 7, Open-Source Near-CDP
Covers three near continuous data protection (near-CDP) products, including
rsync
with snapshots,rsnapshot
, andrdiff-backup
.
Part III
If you have outgrown the capabilities of free utilities or would just like to take advantage of new backup and recovery technologies, you’ll need to look at a commercial product. You should also know about the latest hardware that is on the market to assess your full range of backup and recovery options.
- Chapter 8, Commercial Backup Utilities
Is your guide to the hundreds of features available in the over 50 commercial backup products available on the market today, allowing you to make an educated purchase decision.
- Chapter 9, Backup Hardware
Explains the many different types of backup hardware available today, and provides criteria to help you decide which type of backup drive is right for you.
Part IV
A bare-metal recovery is the fastest way to bring a dead system back to life, even if its operating system drive is completely destroyed.
- Chapter 10, Solaris Bare-Metal Recovery
Explains Sun’s flash archive product, which is the Solaris equivalent of AIX’s
mksysb
.- Chapter 11, Linux and Windows
Explains a number of procedures and tools that can be used to perform bare-metal recovery of both Linux and Windows systems. It includes a discussion of Ghost for Linux (G4L) an open-source ghosting product.
- Chapter 12, HP-UX Bare-Metal Recovery
Covers the
make_net_recovery
andmake_tape_recovery
tools, which now come with HP-UX to perform bare-metal recoveries.- Chapter 13, AIX Bare-Metal Recovery
Discusses AIX’s
mksysb
, probably one of the oldest and best-known bare-metal recovery tools.- Chapter 14, Mac OS X Bare-Metal Recovery
Covers how to perform your own bare-metal recovery of a Mac OS X machine.
Part V
This section explains in plain language an area that presents some of the greatest backup and recovery challenges that a system administrator or database administrator will face—backing up and recovering databases.
- Chapter 15, Backing Up Databases
Explains database architecture while relating each architectural element to the appropriate term in DB2, Exchange, Informix, MySQL, Oracle, PostgreSQL, SQL Server, and Sybase. This chapter will be your friend if you’re an SA who’s afraid of databases or a DBA learning a new database.
- Chapter 16, Oracle Backup and Recovery
Explains how to perform Oracle hot backups using
rman
or user-managed backup.- Chapter 17, Sybase Backup and Recovery
Shows how to use the backup server to back up Sybase ASE.
- Chapter 18, IBM DB2 Backup and Recovery
Explains how to back up and recover DB2 databases.
- Chapter 19, SQL Server
Explains how to back up and recover SQL Server databases.
- Chapter 20, Exchange
Explains how to back up and recover Exchange databases using the built-in
ntbackup
plug-in for Exchange.- Chapter 21, PostgreSQL
Explains how to back up and recover PostgreSQL databases.
- Chapter 22, MySQL
Provides an overview of the various backup and recovery options available for MySQL.
Part VI
The information contained in this part of the book is by no means unimportant; it simply wouldn’t fit anywhere else!
- Chapter 23, VMware and Miscellanea
Includes VMware backups, the oft-debated “live filesystem dumps” question, and even some backup poetry.
- Chapter 24, It’s All About Data Protection
Provides some food for thought, discussing the fact that backups are not the answer to all problems; you should also be thinking about other areas of data protection, such as archiving, disaster recovery, and storage security.
What’s New in This Book
See preceding section. Seriously, this book has about 75 percent new material when compared with Unix Backup & Recovery. Some chapters in the first book were completely rewritten for this book. Here are the highlights of those changes:
- A new philosophy
This book reflects my new backup philosophy, which is that it’s all about disk—especially for smaller shops.
- New backup commands
We’ve added
ntbackup
,ditto
, andrsync
to the basic utilities chapter.- Amanda
The Amanda chapter is completely updated to reflect the developments of the past seven years.
- Commercial utilities
The commercial utilities chapter has been updated to reflect the advances in backup and recovery in the past seven years.
- HP-UX
The
make_net_recovery
andmake_tape_recovery
tools have changed, and so has the chapter covering them.- Backup hardware
Boy, has hardware changed in seven years! Disk targets, virtual tape libraries, and data de-duplication systems. I cover it all.
There are eleven completely new chapters, significantly expanding the scope of this book. Here are the topics covered in these new chapters:
- DB2
How to back up DB2 using its built-in capabilities
- Exchange
How to back up Exchange using
ntbackup
- SQL Server
How to back up SQL Server using its built-in capabilities
- MySQL
How to back up and recover MySQL databases based on the MyISAM, InnoDB, and NDB storage engines
- PostgreSQL
How to back up and recover this popular open-source database using either
pg_dump
orpg_dumpall
- BackupPC
How to use BackupPC, a completely disk-based backup and recovery system with a web frontend
- Bacula
How to use Bacula, an open-source backup product that roams the datacenter at night and sucks the vital essence from your computers
- Near-CDP
How to use snapshots and replication to make backups
- Solaris
How to do bare-metal recovery using flash archive
- Linux and Windows bare-metal recovery
How to use a Linux LiveCD or Ghost for Linux to perform bare-metal recovery of Windows and Linux operating systems
- Mac OS X
How to use the built-in, bare-metal recovery in OS (it isn’t too hard)
What’s Missing?
For various reasons, some chapters from Unix Backup & Recovery did not make it into this book. All of the following chapters are now available online at http://www.backupcentral.com. The one challenge, of course, is that these chapters have not been updated. Therefore, we’ve put them in our wiki so that anyone who wants to help us update them can do so.
Tru64 Bare-Metal Recovery
IRIX Bare-Metal Recovery
Informix Backup and Recovery
Clearcase Backup and Recovery
High Availability
Speaking of BackupCentral.com
We’ve completely redesigned http://www.backupcentral.com using a content management system, forums, and MediaWiki. My number one goal for the new Backup Central is to make it much easier to provide you dynamic content and to build a strong community around backup and recovery issues. The new Backup Central has some really great features:
phpBB forums for various backup-related topics, including one for discussing the book. Come join the discussions.
A mailing list for each forum, allowing you to follow the discussions via the forum or email. Any posts in the forum are sent to the mailing list, and emails sent to the mailing list result in posts or replies in the forum.
A multidirectional connection between backup-related Usenet newsgroups, mailing lists, and phpBB forums. One of the things I was reminded of while writing this book is that Usenet is alive and well, and I want to bring this great resource to the Backup Central community and to create another portal into this underutilized resource. Each relevant Usenet newsgroup has an associated mailing list and forum, and all messages to Usenet, the mailing list, or the forum go to the appropriate forum, mailing list, and newsgroup.
A wiki based on MediaWiki, the same software that drives Wikipedia. One of the things you will find there is a wike entry for every chapter in this book. We’re going to use these entries to update and further the ideas you find in this book. We’re doing this for two reasons:
The first reason is that one of the problems with writing a technical book is that the second you go to press, something changes. While we’re in the process of getting this book printed, MySQL will come out with three more storage engines, QTParted will probably support NTFS, and Bacula’s Windows Server will become generally available. We’ll use the wike to keep things like this up to date.
The second reason is because my contributors and I don’t know all the answers, folks. We did our best to come out with a solid book for you, but we haven’t seen everything you’ve seen. We’d love it if you help us further the ideas mentioned in this book, help us to explain the scenarios under which a given procedure won’t work, or how a given procedure should be enhanced. (For example, right now a friend of mine is trying to help me understand how to get
rsync
to better handle millions of files. His testing won’t be done in time for press. Put it in the Wiki, Jason.) Join the new Backup Central and save the world—or at least its data.
I’ll see you at http://www.backupcentral.com!
Conventions Used in This Book
The following typographical conventions are used in this book:
Constant width
Indicates command-line computer output, computer-generated messages, and code examples. It is also used when referring to commands and parameters in text.
Constant-width italic
Indicates variables in text.
Constant width bold
Indicates user input in command-line examples.
Constant width italic bold
Indicates variables in command-line examples.
- Italic
Introduces new terms and indicates URLs, files, directories, hostnames, and file extensions.
How to Contact Us
We have tested and verified all the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing to:
O’Reilly Media, Inc. |
1005 Gravenstein Highway North |
Sebastopol, CA 95472 |
800-998-9938 (in the United States or Canada) |
707-829-0515 (international/local) |
707-829-0104 (fax) |
We have a web page for the book that lists examples, errata, or any additional information. You can access this page at:
http://www.oreilly.com/catalog/9780596102463/ |
To comment or ask technical questions about this book, send email to:
bookquestions@oreilly.com |
For more information about books, conferences, Resource Centers, and the O’Reilly Network, see the O’Reilly web site at:
http://www.oreilly.com |
O’Reilly Safari®
When you see a Safari® Enabled icon on the cover of your favorite technology book, that means the book is available online through O’Reilly Safari .
Safari offers a solution that’s better than e-books. It’s a virtual library that lets you easily search thousands of top tech books, cut and paste code samples, download chapters, and find quick answers when you need the most accurate, current information. Try it for free at http://oreilly.com/safari.
This Book Was a Team Effort
It’s true; my license plate does say MRBAKUP. But that doesn’t mean I know everything about backup and recovery. In fact, I’ve never even used some of the operating systems or database platforms covered in this book! It would be a disservice to you, the reader, for me to write chapters on those products—but I wanted the chapters in the book. So I hired a team of experts to write the chapters for you. Approximately 250 pages of this book were written by others, and contributors are recognized at the beginning of the chapter(s) they wrote.
Contributors
It’s not an easy thing to write a chapter in someone else’s book. Not only do you have to write, but you have to write based on someone else’s design. There are also tight deadlines, and the process is nothing but hurry up and wait. I couldn’t have done it without them, so please allow me to formally thank all of my contributors.
- Amanda
Contributed by Dmitri Joukovski and Stefan G. Weichinger. Thanks for seeing this one through.
- BackupPC
Contributed by Don “Duck” Harper. Quack!
- Bacula
Contributed by Adam Thornton. Thanks for bringing Bacula to the book.
- Near-CDP
Contributed by Michael Rubel, Ben Escoto, and David Cantrell. This chapter morphed a few times, and I appreciate your patience as it gelled in my head.
- AIX bare-metal recovery
Contributed by Mark Perino. I think you are the fastest writer on the team.
- HP-UX bare-metal recovery
Contributed by Eric Stahl and Ron Goodwyn. Great collaborative effort, guys.
- Linux and Windows bare-metal recovery
Contributed by Reed Robins. Maybe we can do it this way, or that way, or that way! Did I change the scope of the chapter enough? Thanks.
- Mac OS X bare-metal recovery
Contributed by Leon Towns-von Stauber. Thanks, Leon. I sure am glad Mario told me to give you a call. Your chapter was perfect.
- Solaris bare-metal recovery
Contributed by Aaron Gersztoff. Be careful what you ask for, right, Aaron?
- DB2 backup and recovery
Contributed by Jeff Richardson, Kulvir S. Bhogal, and Kondal Yennaram. You guys all came through in a pinch, and I’m most grateful.
- Exchange backup and recovery
Contributed by Scott Harris. More pictures! Fewer pictures! Make it like this, no make it like that! Isn’t it fun writing for me?
- SQL Server backup and recovery
Contributed by Scott Harris. Look at that, Scooter! You’re the only one who was crazy enough to write two chapters for me. Thanks.
- Sybase backup and recovery
Contributed by Edward Barlow, who updated a chapter originally written by Bryn Smith. Another contributor I couldn’t have done without. Thanks.
- Dump internals
Contributed by David Young. When are you going to move out here with your Mom?
Without these folks, this book would contain substantially less information than what you find here.
Technical Editors
Another group of people I must thank is our technical reviewers, and we had a lot of them! The problem with writing a book with a scope this big is that you need specialists and tech reviewers as well. Because of that, most technical editors reviewed only one or two chapters. I couldn’t have done it without them. I’m sure I’ve miss a few, but here’s my best attempt at listing them all (alphabetically by first name):
Adrin Kow |
Andy Shellam |
Anthony Johnson |
Axel Schwenke |
Ben Garrett |
Brian Eliassen |
Brian Peasland |
Charles Whealton |
Chris Thomas |
Christoph Haas |
Craig Barratt |
D.A. Morgan |
Dana Diederich |
Daniel Callahan |
Dave Mehler |
David Boyd |
Edward Conba |
Eric Gilmore |
Eric Stahl |
Finn Henningsen |
Frank Sweetser |
Greg Lehey |
Ian Gorrie |
Ian Herd |
James Bougor |
Jayesh Thakrar |
Jeff Badger |
Jeff Frost |
Jeff Harbert |
Jeff Richardson |
Jeffrey P. Humes |
John Haight |
John Hurley |
John Madden |
Kern Sibbald |
Kumar Sundaram |
Lenz Grimmer |
Marcel Lans |
Mark D. Powell |
Mark Dawson |
Mark Perino |
Massimiliano Daneri |
Matthew Huff |
Megan Restuccia |
Mike Harrold |
Mohammed Mehdi |
Neal A. Lucier |
Norbert Munkel |
Patrick Matthews |
Paul Muggeridge |
Ralph R. Hirtler |
Rob Worman |
Rodrigo Real |
Satyaprakash Pandey |
Scott Boss |
Scott Harris |
Shane Seymour |
Simon Riggs |
Steve Hanson |
Stewart Smith |
Tammy Bednar |
Todd Toles |
Víctor A. Rodríguez |
Vitalis Jerome |
Wil Coulbourn |
William Cole |
Horror Stories
Also giving this book some flavor are those who contributed horror stories. Even if I couldn’t use your story in the book, I want to thank you for sending one in.
Brian O’Neill |
Brian Sakovitch |
Chris Pritchard |
David Bregman |
David J. Young |
Harry Tirrell |
Hywel Matthews |
Jack Coats |
James Hunt |
Jason Frankovitz |
Jason Shupe |
Jim Damoulakis |
Jim Donnellan |
John Merryman |
Jorgen Lie |
Karl Langdon |
Kevin Suttle |
Mark Perino |
Michael Rice |
Michael Tobin |
Natalie Meek |
Richard Ackerman |
Scott Boss |
Theo Van Dinter |
William Birch |
William S. Duncanson |
Special Mention
There were a few people who were extremely helpful in one way or another throughout this project. I’d like to send a special thank you to them.
- Anthony Johnson
It’s not every CEO who would volunteer to tech review a chapter that is essentially a free competitor with his own product. I hope you and Storix do very well. You’ve got quite the Linux and AIX bare-metal recovery tool there.
- Brian Peasland
You beat the living crap out of the first draft of the Oracle chapter, and rightfully so. The new chapter is significantly better thanks to your thorough and honest review. (You made me rewrite half of it, dude!)
- Deb Cameron
Isn’t it fun editing a book with 18 authors from 3 continents, several time zones, a few different native languages, using about 60 technical editors? Let’s do it again sometime!
- Joshua D. Drake
Thanks, Joshua for spending the time you did with me on the phone to help me better understand PostgreSQL. Next time, you should be more forthright with your own opinions. A guy really doesn’t know where he stands with you.
- Lenz Grimmer
You were my liaison in the MySQL community, and I definitely needed the help. Thanks to you and the whole MySQL team.
- Lynn Stone
Thank you for helping get this project off the ground. I couldn’t have done it without you. Only I know your secret identity.
- Tammy Bednar
For such a busy woman, you gave me exactly what I needed for the Oracle part of this project. You’ve obviously done a lot of work on the Oracle backup products, and it shows. I hope you’ll see my newfound respect for your products in the Oracle chapter.
- Zmanda
Thanks for the support on the Amanda chapter, and for bringing commercial support to a very popular open-source backup tool.
- Mario Obejas
Thank you so much for referring me to Leon.
I Don’t Know It All
If there’s one thing I learned while writing this book, it’s that I do not know everything there is to know about backups. If you have a better way to do anything described in this book, have learned any special tricks, or have written any neat utilities that you think would help other people do backups and recoveries, let me know. Email me at curtis@backupcentral.com. Your tricks or utilities may be included in the next edition of the book and listed immediately on http://www.backupcentral.com.
How Can I Say Thanks?
How can I begin to thank the hundreds of people who helped me?
To God: May any praise for this book go to You alone.
To my wife, Celynn: I say “thank you” for the many nights you spent alone while I pounded away at my keyboard somewhere around the globe. You’re a special woman who never gave up on me or my dream. I love you.
To my daughter, Nina: You were only seven when the first book came out. Now you’re a beautiful young lady who is growing up so fast. I’m going to have to get a gun and sit on the porch.
To my daughter, Marissa: You were only two when the first book came out. Now you’re a beautiful nine-year old—my, how time has flown. Let’s go to the park and ride our bikes together.
To my parents: What can I say? You always believed in me. You always used to tell me, “I don’t care if you’re a ditch digger. Just be the best darn ditch digger in the world.” Well, being a backup guy is as close as you can get to being a ditch digger in the computer business, and I “wrote the book” on that.
To Bob Walker for helping me get my first job in backups, and Ron Rodriguez for being all too eager to give it to me.
To Susan Davidson, who didn’t fire me when I couldn’t recover that purchasing database in 1992: that second chance was all I needed to become the expert in backup that I am today. If you had fired me (and I’m sure a few people wanted you to), who knows where I’d be today. (If you’re curious about the story, look for the sidebar “The One That Got Away” in Chapter 1.)
To Collective Technologies for helping me round out my skills enough to see that I wanted to specialize in backup and recovery, and for supporting me when I wrote the first book.
To Jason Stege, Robin Young, Jeff Williams, Reed Robins, and Elia Harris: Thanks for believing in me when I started my own company. I hope I did right by you.
To Mark Shirman and all my friends at GlassHouse: Thanks for giving me a place where I finally feel like I’m using my talents.
To my wife’s family: Thank you for raising such a wonderful lady. Thank you for treating me as one of your own and supporting us on our quest. Pahingi ng sinagang?
To all the teachers who kept trying to get me to live up to my potential: You finally got through.
To O’Reilly: Thank you for the opportunity to bring this much-needed book to market.
To Deb Cameron and Michael Loukides, my editors: We’ll have to actually meet one of these days! I don’t know how you do this, reading the same book over and over, without letting your eyes just glaze over. You’re great editors, and I could really tell that you put your all into this project. Thank you, thank you, and thank you. (Now don’t edit that sentence, OK?)
To the reader: Thank you for purchasing this book. I hope you learn as much reading it as I did writing it.
Get Backup & Recovery now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.