Network Warrior, 2nd Edition

Multiple Component Failure

Many networks are designed to avoid single points of failure. What usually brings these networks down is multiple component failure. Multiple component failure can be triggered by a root cause such as dirty or unreliable power, or it can just be a fluke. One device can also sometimes cause failures in other devices.

Many enterprise networks employ Cisco PIX or ASA firewalls in redundant pairs. This is a reliable and proven way to ensure that your firewall isn’t a single point of failure. If the standby firewall fails and nobody knows about it, the remaining firewall is a single point of failure. The problem is that many people install these failover pairs and then don’t monitor them in any way. I once had a client whose secondary firewall had been offline for months. When the primary failed, the entire network went offline and the executives started screaming. Since no one had been checking the firewalls, everyone assumed that all was well. Nothing sells network management tools like a previously unknown failure causing a system-wide outage. Some of the biggest outages I’ve witnessed have been a result of an offline secondary system failing to back up a primary when it failed.

Sometimes, multiple devices fail for reasons known only to them. In one installation, I saw a dual-supervisor 6509 have a compound failure in which the primary supervisor failed, but the primary MSFC stayed active. Because the MSFC is tied physically to the supervisor to get connectivity ...