Using SANs and NAS | O'Reilly Media

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Throughout the history of computing, people have wanted to share computing resources. The Burroughs Corporation had this in mind in 1961 when they developed multiprogramming and virtual memory. Shugart Associates felt that people would be interested in a way to easily use and share disk devices. That's why they defined the Shugart Associates System Interface (SASI) in 1979. This, of course, was the predecessor to SCSI—the Small Computer System Interface. In the early 1980s, a team of engineers at Sun Microsystems felt that people needed a better way to share files, so they developed NFS. Sun released it to the public in 1984, and it became the Unix community's prevalent method of sharing filesystems. Also in 1984, Sytec developed NetBIOS for IBM; NetBIOS would become the foundation for the SMB protocol that would ultimately become CIFS, the predominant method of sharing files in a Windows environment.

Neither storage area networks (SANs) nor network attached storage (NAS) are new concepts. SANs are simply the next evolution of SCSI, and NAS is the next evolution of NFS and CIFS. Perhaps a brief history lesson will illustrate this (see Figure 1-1).

Figure 1-1: Generations of SCSI

As mentioned earlier, SCSI has its origins in SASI—the Shugart Associates System Interface, defined by Shugart Associates in 1979. In 1981, Shugart and NCR joined forces to better document SASI and to add features from another interface developed by NCR. In 1982, the ANSI task group X3T9.3 drafted a formal proposal for the Small Computer System Interface (SCSI), which was to be based on SASI. After work by many companies and many people, SCSI became a formal ANSI standard in 1986. Shortly thereafter, work began on SCSI-2, which incorporated the Common Command Set into SCSI, as well as other enhancements. It was approved in July 1990.

Although SCSI-2 became the de facto interface between storage devices and small to midrange computing devices, not everyone felt that traditional SCSI was a good idea. This was due to the physical and electrical characteristics of copper-based parallel SCSI cables. (SCSI systems based on such cables are now referred to as parallel S

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

As mentioned earlier, SCSI has its origins in SASI—the Shugart Associates System Interface, defined by Shugart Associates in 1979. In 1981, Shugart and NCR joined forces to better document SASI and to add features from another interface developed by NCR. In 1982, the ANSI task group X3T9.3 drafted a formal proposal for the Small Computer System Interface (SCSI), which was to be based on SASI. After work by many companies and many people, SCSI became a formal ANSI standard in 1986. Shortly thereafter, work began on SCSI-2, which incorporated the Common Command Set into SCSI, as well as other enhancements. It was approved in July 1990.

Although SCSI-2 became the de facto interface between storage devices and small to midrange computing devices, not everyone felt that traditional SCSI was a good idea. This was due to the physical and electrical characteristics of copper-based parallel SCSI cables. (SCSI systems based on such cables are now referred to as parallel SCSI, because the SCSI signals are carried across dozens of pairs of conductors in parallel.) Although SCSI has come a long way since 1990, the following limitations still apply to parallel SCSI:

Parallel SCSI is limited to 16 devices on a bus.
It's possible, but not usually practical, to connect two computing devices to the same storage device with parallel SCSI.
Due to cross-talk between the individual conductors in a multiconductor parallel SCSI cable, as well as electrical interference from external sources, parallel SCSI has cable-length limitations. Although this limitation has been somewhat overcome by SCSI-to-fiber-to-SCSI conversion boxes, these boxes aren't supported by many software and hardware vendors.
It's also important to note that each device added to a SCSI chain shortens its total possible length.

Some felt that in order to solve these problems, we needed to change the physical layer. The most obvious answer at the time was fiber optics. Unlike parallel SCSI, fiber cables can go hundreds of meters without significantly changing their transmission characteristics, solving all the problems related to the electrical characteristics of parallel SCSI. It even solved the problem of the number of connections, since each device in the loop had its own transmitting laser. Therefore, additional devices actually increase the total bus length, rather than shorten it.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

At this point, my definition of a SAN is as follows:

A SAN is two or more devices communicating via a serial SCSI protocol, such as Fibre Channel or iSCSI.

This definition means that a SAN isn't a lot of things. By this definition, a LAN that carries nothing but storage traffic isn't a SAN. What differentiates a SAN from a LAN (or from NAS) is the protocol that is used. If a LAN carries storage traffic using the iSCSI protocol, then I'd consider it a SAN. But simply sending traditional, LAN-based backups across a dedicated LAN doesn't make that LAN a SAN. Although some people refer to such a network as a storage area network, I do not, and I find doing so very confusing. Such a network is nothing other than a LAN dedicated to a special purpose. I usually refer to this sort of LAN as a "storage LAN" or a "backup network." A storage LAN is a useful tool that removes storage traffic from the production LAN. A SAN is a network that uses a serial SCSI protocol (e.g., Fibre Channel or iSCSI) to transfer data.

A SAN isn'tnetwork attached storage (NAS). As mentioned previously, SANs use the SCSI protocol, and NAS uses the NFS and SMB/CIFS protocols. (There will be a more detailed comparison between SANs and NAS at the conclusion of this chapter.) The Direct Access File System, or DAFS, pledges to bring SANs and NAS closer together by supporting file sharing via an NFS-like protocol that will also support Fibre Channel as a transport. (DAFS is covered in Appendix A.)

It's common for a NAS filer to be comprised of a filer head with SAN-attached storage behind it.

In summary, a SAN is two or more devices communicating via a serial SCSI protocol (e.g., Fibre Channel or iSCSI), and they offer a number of advantages over traditional, parallel SCSI:

Fibre Channel (and iSCSI) can be trunked, where several connections are seen as one, allowing them to communicate much faster than parallel SCSI. Even a single Fibre Channel connection now runs at 2 Gb/s in each direction, for a total aggregate of 4 Gb/s.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A long time ago in a data center far away, there were servers that were small enough to fit on a tape. This type of data center led to a backup system design like the one in Figure 1-3. Many or most systems came with their own tape drive, and that tape drive was big enough to back up that system—possibly big enough to back up other systems. All that was needed to perform a fully automated backup was to write a few shell scripts and swap out a few tapes in the morning.

Figure 1-3: Backups in the good old days

For several reasons, bandwidth was not a problem in those days. The first reason was there just wasn't that much data to back up. Even if the environment consisted of a single 10-Mb hub that was chock full of collisions, there just wasn't that much data to send across the wire. The second reason that bandwidth wasn't a problem was that many of the systems could afford to have their own tape drives, so there wasn't a need to send any data across the LAN.

Gradually, many companies or individuals began to outgrow these systems. Either they got tired of swapping that many tapes, or they had systems that wouldn't fit on a tape any more. The industry needed to come up with something better.

A few early innovators came up with the concept of a centralized backup server. Combining this with a tape stacker made life manageable again. Now all you had to do was spend $5,000 to $10,000 on backup software and $5,000 to $10,000 on hardware, and your problems were solved. Every one of your systems would be backed up across the network to the central backup server, and all you needed to do was install the appropriate piece of software on each backup "client." These software packages even ported their client software to many different platforms, which meant that all the systems shown in Figure 1-4 could be backed up to the backup server, regardless of what operating system they were running.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

NAS has roots in two main areas: the Server Message Block (SMB) and the Network File System (NFS) protocols. Interestingly enough, these two file-sharing protocols were developed by two of the biggest rivals in the computer industry: Microsoft and Sun Microsystems, and they both appeared in 1984! (IBM and Microsoft developed SMB, and Sun developed NFS.)

When you use your Windows browser to go to "Network Neighborhood" and can read from and write to drives that are actually on other computers, you are probably using the SMB protocol. The first mention of the SMB protocol was in an IBM technical reference in 1984, and it was originally designed to be a network naming and browsing protocol. Shortly thereafter, it was adapted by Microsoft to become a file sharing protocol. Several versions of the SMB protocol have been released throughout the years, and it has become the common file sharing protocol for all Microsoft Windows operating systems (Windows 3.1, 95, 98, Me, NT, 2000, and XP) and IBM OS/2 systems. Microsoft recently changed its name to the Common Internet File System (CIFS).

Like many Microsoft applications, CIFS was designed for simplicity. To allow others to access a drive on your system, simply right-click on a drive icon and select "Sharing." You then decide whether the drive should be shared read-only or read-write, and what passwords should control access. You can share a complete drive (e.g., C:\) or just a part of the drive (e.g., C:\MYMUSIC).

Please note that I didn't say that CIFS was originally designed for performance. Actually, as covered later in the book, it was designed with multiple-user access in mind—at the expense of performance. However, Microsoft and other companies have made a number of performance improvements to CIFS in recent years.

The popularity of CIFS has led to many companies installing large, centralized CIFS servers that share drives to hundreds or thousands of PC clients. The most common reason to do this is to centralize the storage of important files. Users are encouraged to save anything important on the "network drive" because many don't back up their desktops. The administration staff, however, backs up the CIFS server.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Table 1-1 compares NAS and SAN and should clear up any confusion regarding their similarities and differences.

Table 1-1: Differences between SAN and NAS
	SAN	NAS
Protocol	Serial SCSI-3	NFS/CIFS
Shares	Raw disk and tape drives	Filesystems
Examples of shared items	/dev/rmt/0cbn /dev/dsk/c0t0d0s2 \\.\Tape0	\\filer\C\directory\filename.doc /nfsmount/directory/filename.txt
Allows	Different servers can access the same raw disk or tape drive (not typically seen by the end user)	Different users can access the same filesystem or file
Replaces	Replaces locally attached disk and tape drives; with SANs, hundreds of systems can now share the same disk or tape drive	Replaces Unix NFS servers and NT CIFS servers that offer network shared filesystems

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

One reason this book includes both SANs and NAS is that many people are starting to see filers as a viable alternative to a SAN. While filers were once perceived as "NFS in a box," people are now using them to host large databases and important production data. Which is right for you? The last section of this chapter attempts to explain the pros and cons of each architecture, allowing you to answer this question.

Many comments made in the following paragraphs are summaries of statements made in later chapters. For the details behind these summary statements, please read Chapter 2 and Chapter 5.

As mentioned earlier, NAS filers have become popular with many people for many reasons. The following is a summary of several:

Filers are fast enough for many applications: Many would argue that SANs are simply more powerful than NAS. Some would argue that NFS and CIFS running on top of TCP/IP creates more overhead on the client than SCSI-3 running on top of Fibre Channel. This would mean that a single host could sustain more throughput to a SAN-based disk than a NAS-based disk. While this may be true on very high-end servers, most real-world applications require much less throughput than the maximum available throughput of a filer.
NAS offers multihost filesystem access: A downside of SANs is that, while they do offer multihost access to devices, most applications want multihost access to files. If you want the systems connected to a SAN to read and write to the same file, you need a SAN or cluster-based filesystem. Such filesystems are starting to become available, but they are usually expensive and are relatively new technologies. Filers, on the other hand, offer multihost access to files using technology that has existed since 1984.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Before jumping headfirst into using storage area networks, you need to understand something about the technology upon which most of them are based: Fibre Channel. You need to know answers to questions such as:

Just what is Fibre Channel, anyway?
What are the different variations (topologies) of Fibre Channel?
What are the different parts of a Fibre Channel SAN?

Although this chapter is limited in size, it should explain Fibre Channel enough that you can understand the rest of the SAN chapters in this book. If you need to learn more about the inner workings of Fibre Channel, there are plenty of good books to help you.

SANs certainly qualify as leading-edge technology. Some things being done with SANs can even be considered "bleeding edge." It's no surprise, then, that many people don't fully understand them yet. However, Fibre Channel has been around for years, and it is also a mystery. Why is that? Perhaps the reason Fibre Channel is so unknown is that you really didn't need to know too much about it up to this point. For most people, Fibre Channel meant plugging in a Fibre Channel disk to a host, and that was the end of it. They didn't know that they were creating a point-to-point Fibre Channel network when they did that. Even more people didn't realize that they were creating an arbitrated loop when they daisy-chained a few storage arrays together on the back of a server.

You don't need to know a whole lot about Fibre Channel when using it on such a small network. However, if you're trying to build even a reasonably sized SAN, not knowing Fibre Channel can greatly increase your confusion, frustration, and the total cost of the project.

As covered in Chapter 1, the purpose of Fibre Channel was to remove the performance and logistical barriers of legacy LANs and the parallel SCSI architecture. The features of Fibre Channel include:

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

SANs certainly qualify as leading-edge technology. Some things being done with SANs can even be considered "bleeding edge." It's no surprise, then, that many people don't fully understand them yet. However, Fibre Channel has been around for years, and it is also a mystery. Why is that? Perhaps the reason Fibre Channel is so unknown is that you really didn't need to know too much about it up to this point. For most people, Fibre Channel meant plugging in a Fibre Channel disk to a host, and that was the end of it. They didn't know that they were creating a point-to-point Fibre Channel network when they did that. Even more people didn't realize that they were creating an arbitrated loop when they daisy-chained a few storage arrays together on the back of a server.

You don't need to know a whole lot about Fibre Channel when using it on such a small network. However, if you're trying to build even a reasonably sized SAN, not knowing Fibre Channel can greatly increase your confusion, frustration, and the total cost of the project.

As covered in Chapter 1, the purpose of Fibre Channel was to remove the performance and logistical barriers of legacy LANs and the parallel SCSI architecture. The features of Fibre Channel include:

Support for other, typically "non-network" protocols, such as SCSI. (This will be important for our discussion.)
Confirmed delivery, thereby enhancing the reliability of the network.
True quality of service (QOS) features, including fractional bandwidth and connection-oriented virtual circuits to guarantee bandwidth for critical backups or other operations.
Extremely low-latency connection and connectionless service.
Support for three different network topologies (point-to-point, arbitrated loop, and fabric), including auto-discovery of each topology and of all nodes placed on the network.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Before I explain the different Fibre Channel topologies, you need to know the names of the different types of Fibre Channel ports that are used in those topologies. There are three basic types of ports: the N_Port, the F_Port, and the E_Port. Then as you add arbitrated loop capabilities to these basic ports, they take the combined names of NL_Port, FL_Port, and G_Port.

N_Port: An N_Port is a node port, or a port on a disk or computer. If a port is onlyan N_Port (and not an NL_Port), it can communicate only with another N_Port on a second node or to an F_Port on a switch.
F_Port: An F_Port is a fabric port, which is found only on a switch. If a port is onlyan F_Port (and not an FL_Port), it can connect only to another N_Port via a point-to-point connection.
L_Port: An L_Port implies it can participate in an arbitrated loop. If a port was only an L_Port, it could connect only to arbitrated loops, but ports that are exclusively L_Ports don't exist. The L is added to the end of an N_Port or F_Port to create an NL_Portor an FL_Port.
NL_Port: A node port with arbitrated loop capabilities; that is, a port on a node that can connect to another node, to a switch (see N_Port) or to an arbitrated loop (see the L_Portdefinition).
FL_Port: A fabric port with arbitrated loop capabilities; that is, a port on a switch that can connect to a node (see F_Port) or an arbitrated loop (see L_Port).
E_Port: An E_Port is an expansion port on a switch that connects one switch to other switches via their E_Ports to form a large fabric.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

There are three Fibre Channel network topologies, the simplest and least expensive of which is point-to-point. The most expensive and complex Fibre Channel topology is the fabric topology, but it also has the greatest amount of functionality. The remaining topology is arbitrated loop, which fits right between point-to-point and fabric with regards to cost and functionality.

A point-to-point Fibre Channel network is the simplest and least expensive of the three topologies, and is simply two N_Ports communicating via a point-to-point connection. As seen in Figure 2-2, a Fibre Channel array connected to a host is an example of a point-to-point connection.

Figure 2-2: Point-to-point topology

An illustration of a basic fabric network can be found in Figure 2-3. In a true fabric-only environment, each N_Port plugs into one F_Port on the switch. Each node is then assigned its native address identifier by the switch when it "logs into" the fabric. (This is why this topology is sometimes referred to as "fabric login.") This 24-bit address allows for up to 16 million unique addresses within a single fabric, which should be enough for even the biggest SANs. (At this point, I can't possibly imagine a SAN with 16 million nodes on it, but then I keep thinking about what the popularity of the Internet did to the original IP specification.)

Figure 2-3: Fabric topology

When an N_Port is connected to a switch, it and other N_Ports connected to that switch can use the entire bandwidth of the port to which they are attached. Just as four nodes in a 100-Mb switched Ethernet environment can hold two simultaneous 100-Mb/s "conversations," every port in a Fibre Channel switch can supply the connected node with as much bandwidth as it needs—up to 100 MB/s. (That is, if the switch has been designed with the proper internal bandwidth. Not all switches are created equal.)

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Let's now discuss the elements of a SAN and how they work together. The main elements of a SAN are servers, HBAs, switches, hubs, routers, disk systems, tape systems, cabling, and software. These are all illustrated in Figure 2-9.

Figure 2-9: Elements of a SAN

No storage area network would have any reason for being if there weren't servers connected to it. The servers use the SAN to share storage resources. These servers can be anything from a traditional server to a high-end graphics workstation.

Servers connect to the SAN via their host bus adapter. This is often referred to as a "Fibre Channel card" or "Fibre Channel NIC." It's simply the Fibre Channel equivalent of a SCSI card (i.e., a SCSI HBA). As I've already mentioned, some HBAs may use fiber, and some HBAs may use copper. Regardless of the physical layer, the HBA connects the servers to the SAN.

An iSCSI HBA is a standard, Ethernet NIC whose drivers have been updated to allow the transmission of serial SCSI-3 data across IP. Although theoretically this can be done with any Ethernet NIC, it's usually done with newer, hardware-accelerated Gigabit Ethernet NICs.

Many people look at the networks depicted here and say, "What's the difference between a LAN and a SAN?" The answer is really easy: the protocol they use. Systems on the LAN use IP, IPX, and other "typical" network protocols to communicate with other systems. Systems on the SAN use the SCSI-3 protocol (typically sent over Fibre Channel) to communicate with storage devices.

The term "SCSI" can be a bit confusing because the acronym refers to both the physical medium (a parallel SCSI cable) and the protocol that carries the traffic. Perhaps a comparison will help. IP traffic is sent over copper twisted-pair cables and fiber optic cables via the Ethernet protocol. SCSI traffic is sent over copper SCSI cables via the SCSI protocol. In this case, the SCSI protocol is performing essentially the same duties as the IP andEthernet protocols. SCSI traffic also travels over fiber via the SCSI-3 protocol running on top of the Fibre Channel protocol. In this case, the SCSI-3 protocol is performing essentially the same duties as IP, and the Fibre Channel protocol is performing essentially the same duties as the Ethernet protocol.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

To share resources, you need a network. To share computing resources, the industry developed the LAN, which enabled access to any computing resource (computer) from any other computing resource. This introduced the concept of a shared resource, in which one computing resource could be shared by multiple clients.

Just as the LAN allowed the introduction of NFS (a shared filesystem that was a revolutionary concept when it first appeared), the SAN has introduced a new concept: a shared storage device. A shared storage device is a raw storage device (such as a disk, optical, or tape drive) that is connected to a SAN and appears as a locally attached device to any computer connected to the SAN. To put it in simpler terms, connecting a tape drive to a SAN allows any computer attached to that SAN to perform a backup to that tape drive just as if that tape drive were physically attached to that computer via a SCSI cable. And with this wonderfully new concept, we can accomplish wonderful things.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

What's it like to manage a SAN? What kind of applications lend themselves to SANs? What kind of tasks do you find yourself performing once you've decided to actually purchase a SAN? Why does a SAN require more management than traditional, parallel SCSI? What can you do with a SAN that you can't do without a SAN? These are all important questions.

Many of the reasons you would want to use a SAN are the same as the reasons for using NAS. Those reasons are covered primarily in Chapter 1. Chapter 6 covers the applications where NAS systems perform exceptionally well. Therefore, this section of this chapter covers only the applications in which SANs tend to perform better than NAS.

SANs perform well with large, high-performance databases. While some databases support placing their datafiles on a NAS filer, many don't. Even if the database doesn't specifically support NAS, there may be performance issues with running a large, high-performance database on NAS. Therefore, many environments chose to run their databases on SAN-attached disks. SAN-attached disks offer two distinct advantages to NAS filers:

High-performance backups: When you talk about terabyte-sized databases, it's hard to beat the backup and recovery performance of SAN-attached disks, especially if you start talking about server- or client-free backups. Such backups allow you to back up a large amount of data in a short period of time, without any performance impact to the application. There are parallels to some of these backup types in the NAS world, but SAN-based backups are almost always faster than NAS-based backups. Whether or not you need that additional speed is your decision.
Faster database performance: If your budget is unlimited, you can't beat SAN-attached disks for performance. If cost is a factor, though, some NAS systems may actually be faster than equivalently priced SAN systems. But, if you've got the money, SAN-attached disks will be fastest.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Many of the reasons you would want to use a SAN are the same as the reasons for using NAS. Those reasons are covered primarily in Chapter 1. Chapter 6 covers the applications where NAS systems perform exceptionally well. Therefore, this section of this chapter covers only the applications in which SANs tend to perform better than NAS.

SANs perform well with large, high-performance databases. While some databases support placing their datafiles on a NAS filer, many don't. Even if the database doesn't specifically support NAS, there may be performance issues with running a large, high-performance database on NAS. Therefore, many environments chose to run their databases on SAN-attached disks. SAN-attached disks offer two distinct advantages to NAS filers:

High-performance backups: When you talk about terabyte-sized databases, it's hard to beat the backup and recovery performance of SAN-attached disks, especially if you start talking about server- or client-free backups. Such backups allow you to back up a large amount of data in a short period of time, without any performance impact to the application. There are parallels to some of these backup types in the NAS world, but SAN-based backups are almost always faster than NAS-based backups. Whether or not you need that additional speed is your decision.
Faster database performance: If your budget is unlimited, you can't beat SAN-attached disks for performance. If cost is a factor, though, some NAS systems may actually be faster than equivalently priced SAN systems. But, if you've got the money, SAN-attached disks will be fastest.

There are some applications that generate hundreds of thousands, or even millions of files—even within the same filesystem. Why is this an issue? The issue is that it's almost impossible to back up such a filesystem via standard, filesystem-based backup software. This is why most of the major backup software vendors have developed image-level backup and recovery for such filesystems. This allows them to back up the filesystem via its raw device. Recovery can be done at the file level for individual files or at the raw device level for high-performance recoveries.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

There is no question that SANs bring significant advancements to the storage industry. However, taking advantage of this technology requires a major shift in how you think about and manage your storage. On one hand, Fibre Channel, the technology on which most SANs are based, is simply a new physical layer upon which SCSI data can travel. On the other hand, Fibre Channel has broken the SCSI mold in so many ways that it challenges a number of assumptions many of us have had for years about our SCSI-based storage devices. Since some of these assumptions are no longer true, it creates new issues that must be addressed—issues that must be managed.

Access to storage resources in a Fibre Channel network is accomplished using channels (thus the name Fibre Channel) through a loop (arbitrated loop), or a fabric (a network of SAN fabric switches) to connect one or more hosts to one or more storage devices. Each individual channel, or data path, in a SAN provides a virtual end-to-end connection between a server or host through the SAN to any intended and assigned storage resource, including all physical components and logical connections. Each path can be thought of, both physically and logically, as a virtual representation of a singular SCSI connection or cable. However, unlike the SCSI cable, this cable can create a channel from any SAN-connected server to any SAN-connected storage device. This results in many interesting differences between SANs and parallel SCSI.

iSCSI SANs will have all of these issues as well.

The first assumption about SCSI-based storage that isn't true with SANs is that each SCSI device will appear only on one SCSI bus. With SANs, it's perfectly normal to connect more than one Fibre Channel HBA on the same host to the same SAN, resulting in multiple paths to the same device. Since switches and some storage devices can also accept multiple Fibre Channel connections, the number of physical paths to each device may actually be quite high. These paths are managed by a routing protocol on the switches, and software running on the operating system of each client using the SAN.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

If all your SAN devices are functioning properly, you will be spending much of your SAN management time allocating storage resources and giving access to them. First, you will create virtual disk and tape devices through a process called virtualization . Second, you will create zones or use LUN masking to create data paths between the virtual storage resources and the servers that need them. Third, you can use persistent binding to create a permanent relationship between each virtual resource and a SCSI ID on a server. Finally, you use multipathing software to take advantage of the multiple physical paths between each storage resource and each server.

A SAN may consist of hundreds or thousands of physical disks inside one or more storage arrays, and many tape drives inside one or more tape libraries. The first task of the SAN administrator is to turn these physical resources into virtual ones that may be used by the servers. This is done by dividing a single physical resource into multiple resources (i.e., slicing), or by aggregating many physical resources into one virtual resource (i.e., striping/RAID ). There isn't much to do with tape libraries, because you can't create slices of a tape drive or stripe many tape drives together. However, with disk arrays, there's quite a bit of work to do, and the virtualization of disk storage can take place on either the storage array or the server level.

Section 3.3.1.1: Slicing

All storage consists of a number of individual disks. These disks can be treated as complete units or can be sliced into smaller sections. One reason for doing this is the performance characteristics of the various sections of the disk. Each disk is made of several platters, and each platter contains three sections, or bands, each with different performance characteristics. This is because the speed at which the read/write head is passing over the data is greater when it's closer to the outer edge of the platter, and lesser when it's closer to the inner edge of the platter.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Once the SAN is installed, configured, and doing what you designed it to do, things get much better. Even so, there's still plenty of work to be done. The rest of the work involves managing, monitoring, and maintaining the storage.

Few SANs live on using their initial design. Even the one shown in Figure 3-4 can be outgrown with time. At some point, someone will want to change the SAN in order to:

Increase or decrease the number of servers: If a SAN is successful, it becomes popular. Everybody wants to put their storage on the SAN. This results in more HBAs, more ports, and more switches. Another common occurrence in SANs is no different than parallel based SCSI: somebody gets the bright idea of consolidating servers. This, of course, changes the design of the SAN.
Increase or decrease the number of storage arrays: Whether you're adding more servers to the SAN or not, the need for additional storage capacity will inevitably drive you to purchase more storage. Will you buy additional storage arrays of the size you are using, or will you look at larger, more centralized storage arrays? Even if you don't grow your storage dramatically, what about centralizing all your storage into one or more large storage arrays anyway? What affect will these changes have on the number of ports and switches required for your SAN?
Increase or decrease the number of switches, hubs, or routers: This change, of course, is driven by the previous two changes. Whether you increase or decrease the number of servers or storage arrays, it will change the number of ports that you need. Will you buy additional switches and use inter-switch links, or ISLs? As you gain experience with your SAN, you will probably also want to increase its level of availability. What about purchasing a director class switch? What does the SAN look like now that you've changed it?

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Although they do create additional management issues, SANS can allow you to maximize your investment in both online (disk) and offline (tape and optical) storage.

When speaking rather broadly, there are essentially three things the SAN in Figure 3-5 makes possible:

Online storage maximization
Offline storage maximization
Truly highly available systems

Figure 3-5: Elements of a SAN

Online storage maximization allows you to create a pool of disk systems that can be allocated to the servers that need them—as well as many other things. Because a SAN allows for multiple servers to access multiple physical disks via multiple physical paths, it also allows for truly highly available systems. Offline storage maximization means being able to configure your backup and recovery system in ways that you were never able to do before.

Many large data centers have thousands of discrete disks between all of their servers. There are a lot of disadvantages to this that can be overcome by SANs.

When one disk fails, the server often has to be taken down in order to repair it. While this can be solved by buying RAID arrays with hot-swappable disks and at least one hot spare for each array, allocating a hot spare for each array can be expensive. Also, if the hot spare has already been used and not been replaced, you can't automatically use the spare from one server to fix a bad drive in another server. You wouldn't even want to do this manually, since doing so degrades the integrity of the server you borrowed the drive from.

Consider aging or decrepit disk drives. I remember one client that had an entire set of servers that could not afford down time. Behind these servers were disk drives with a known problem that wasn't discovered until the servers had been in production for a long time. They would occasionally leak swag oil on the disk, rendering it useless. The vendor was offering free replacements, but doing so took over a year due to the downtime and hassle required to make the swap. Although this is similar to the last problem, it's slightly different. Using discrete disks doesn't easily allow for preventive replacement of drives you know are old or potentially faulty.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Don't assume that the SAN will manage itself, and you'll be doing better than most! Besides having a good initial design and implementation of your SAN, the best thing you can do to make life in your SAN-box better is to evaluate and select a good storage resource management product, then test it over and over before you put it in production. Also, having a SAN can also allow you to do some really wonderful things, including online and offline storage management and truly highly available systems.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A SAN typically consists of multiple servers, online storage (disk), and offline storage (tape or optical), all of which are connected to a Fibre Channel switch or hub—usually a switch. You can see all these SAN elements in Figure 4-1. Once the three servers in Figure 4-1 are connected to the SAN, each server in the SAN can be granted full read/write access to any disk or tape drive within the SAN. This allows for LAN-free, client-free, and server-free backups, each represented by a different-numbered arrow in Figure 4-1.

Figure 4-1: LAN-free, client-free, and server-free backups

Pretty much anything said in this chapter about Fibre Channel-based SANs will soon be true about iSCSI-based SANs. To visualize how an iSCSI SAN fits into the pictures and explanations in this chapter, simply replace the Fibre Channel HBAs with iSCSI-capable NICs, and the Fibre Channel switches and hubs with Ethernet switches and hubs, and you've got yourself an iSCSI-based SAN that works essentially like the Fibre Channel-based SANs discussed in this chapter. The difficulty will be in getting the storage array, library, and software vendors to support it.

LAN-free backups: LAN-free backups occur when several servers share a single tape library. Each server connected to the SAN can back up to tape drives it believes are locally attached. The data is transferred via the SAN using the SCSI-3 protocol, and thus doesn't use the LAN. All that is needed is software that will act as a "traffic cop." LAN-free backups are represented in Figure 4-1 by arrow number 1, which shows a data path starting at the backup client, traveling through the SAN switch and router, finally arriving at the shared tape library.
Client-free backups

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

A SAN typically consists of multiple servers, online storage (disk), and offline storage (tape or optical), all of which are connected to a Fibre Channel switch or hub—usually a switch. You can see all these SAN elements in Figure 4-1. Once the three servers in Figure 4-1 are connected to the SAN, each server in the SAN can be granted full read/write access to any disk or tape drive within the SAN. This allows for LAN-free, client-free, and server-free backups, each represented by a different-numbered arrow in Figure 4-1.

Figure 4-1: LAN-free, client-free, and server-free backups

Pretty much anything said in this chapter about Fibre Channel-based SANs will soon be true about iSCSI-based SANs. To visualize how an iSCSI SAN fits into the pictures and explanations in this chapter, simply replace the Fibre Channel HBAs with iSCSI-capable NICs, and the Fibre Channel switches and hubs with Ethernet switches and hubs, and you've got yourself an iSCSI-based SAN that works essentially like the Fibre Channel-based SANs discussed in this chapter. The difficulty will be in getting the storage array, library, and software vendors to support it.

LAN-free backups: LAN-free backups occur when several servers share a single tape library. Each server connected to the SAN can back up to tape drives it believes are locally attached. The data is transferred via the SAN using the SCSI-3 protocol, and thus doesn't use the LAN. All that is needed is software that will act as a "traffic cop." LAN-free backups are represented in Figure 4-1 by arrow number 1, which shows a data path starting at the backup client, traveling through the SAN switch and router, finally arriving at the shared tape library.
Client-free backups: Although an individual computer is often called a server, it's referred to by the backup system as a

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

As discussed in Chapter 1, LAN-free backups allow you to use a SAN to share one of the most expensive components of your backup and recovery system—your tape or optical library and the drives within it. Figure 4-2 shows how this is simply the latest evolution of centralized backups. There was a time when most backups were done to locally attached tape drives. This method worked fine when data centers were small, and each server could fit on a single tape. Once the management of dozens (or even hundreds) of individual tapes became too much or when servers would no longer fit on a tape, data centers started using backup software that allowed them to use a central backup server and back up their servers across the LAN. (The servers are now referred to by the backup system as clients.)

Figure 4-2: The evolution of backup methodologies

This methodology works great as long as you have a LAN that can support the amount of network traffic such backups generate. Even if you have a state-of-the-art LAN, you may find individual backup clients that are too big to back up across the LAN. Also, increasingly large amounts of system resources are required on the backup server and clients to back up large amounts of data across the LAN. Luckily, backup software companies saw this coming and added support for remote devices. This meant that you could again decentralize your backups by placing tape drives on each backup client. Each client would then be told when and what to back up by the central backup server, but the data would be transferred to a locally attached tape drive. Most major software vendors also allowed this to be done within a tape library. As depicted in Figure 4-2, you can connect one or more tape drives from a tape library to each backup client that needs them. The physical movement of the media within the library is then managed centrally—usually by the backup server.

Although the backup data at the bottom of Figure 4-2 isn't going across the LAN, this isn't typically referred to as LAN-free backups. The configuration depicted at the bottom of Figure 4-2 is normally referred to as

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

Performing client-free backup and recovery requires the coordination of several steps across at least two hosts. At one point in time, none of the popular commercial backup and recovery applications had software that could automate all the steps without requiring custom scripting by the administrator. All early implementations of client-free backups involved a significant amount of scripting on the part of the administrator, and almost all early implementations of client-free backups were on Unix servers. This was for several reasons, the first of which was demand. Many people had really large, multi-terabyte Unix servers that qualified as candidates for client-free backups. This led to a lot of cooperation between the storage array vendors and the Unix vendors, which led to commands that could run on a Unix system and accomplish the tasks required to make client-free backups possible. Since many of these tasks required steps to be coordinated on multiple computers, the rsh and ssh capabilities of Unix came in handy.

Since NT systems lacked integrated, advanced scripting support, and communications between NT machines were easy for administrators to script, it wasn't simple to design a scripted solution for NT client-free backups. (As you will see later in this section, another key component that was missing was the ability to mount brand new drive letters from the command line.) This, combined with the fact that NT machines tended to use less storage than their monolithic Unix counterparts, meant that there were not a lot of early implementations of client-free backup on NT. However, things have changed in recent years. It isn't uncommon to find very large Windows machines. (I personally have seen one approaching a terabyte.) Scripting and intermachine communication has improved in recent years, but the limitation of not being able to mount drives via the command line has existed until just recently. Therefore, it's good that a few commercial backup software companies are beginning to release client-free backup software that includes the Windows platform. Those of us with very large Windows machines can finally take advantage of this technology.

Additional content appearing in this section has been removed.
Purchase this book now or read it online at Safari to get the whole thing!

Content preview·Buy PDF of this chapter|Buy reprint rights for this chapter

So far in this chapter, we have discussed LAN-free and client-free backups. LAN-free backups help speed backup and recovery by removing this traffic from the LAN. They allow you to purchase one or more really large tape libraries and share those tape libraries among multiple servers. This allows you to take advantage of the economies of scale that large libraries provide, as the cost per MB usually decreases as the size of a library increases. Backing up large amounts of data is much easier on the clients when that data is sent via Fibre Channel, instead of being sent across the LAN. Therefore, LAN-free backups reduce the degree to which backups affect any applications that are running on the client.

Client-free backups also offer multiple advantages. They remove almost all application impact from the backup client, since all significant I/O operations are performed on the backup server. If you use a backup mirror for each set of primary disk sets, they also offer a virtually instantaneous recovery method. However, this recovery speed does come with a significant cost, since purchasing a backup mirror for each set of primary disk sets requires purchasing more disk capacity then would otherwise be necessary. If your primary disk set is mirrored, you need to purchase a backup mirror, requiring the purchase of 50% more disk. If the primary disk set is a RAID 4 or RAID 5 volume, you will need to purchase almost 100% more disk.

What about snapshots? You may remember that we have discussed snapshots as an alternate way to provide instantaneous recoveries. However, snapshots (that have not been backed up to tape) only protect against logical corruption, and the loss of too many drives on the primary disk set results in a worthless snapshot. Therefore, client-free backups that use a backup mirror are the only backup and recovery design that offers instantaneous recovery after the loss of multiple drives on the primary disk set.

The remaining disadvantage to client-free backups is that you still need a serve

Table of Contents

Section 3.3.1.1: Slicing