Troubleshooting Cisco Nexus Switches and NX-OS, First Edition

Book description

The definitive deep-dive guide to hardware and software troubleshooting on Cisco Nexus switches

The Cisco Nexus platform and NX-OS switch operating system combine to deliver unprecedented speed, capacity, resilience, and flexibility in today's data center networks. Troubleshooting Cisco Nexus Switches and NX-OS is your single reference for quickly identifying and solving problems with these business-critical technologies.

Three expert authors draw on deep experience with large Cisco customers, emphasizing the most common issues in real-world deployments, including problems that have caused major data center outages. Their authoritative, hands-on guidance addresses both features and architecture, helping you troubleshoot both control plane forwarding and data plane/data path problems and use NX-OS APIs to automate and simplify troubleshooting. Throughout, you'll find real-world configurations, intuitive illustrations, and practical insights into key platform-specific behaviors.

This is an indispensable technical resource for all Cisco network consultants, system/support engineers, network operations professionals, and CCNP/CCIE certification candidates working in the data center domain.

  • Understand the NX-OS operating system and its powerful troubleshooting tools
  • Solve problems with cards, hardware drops, fabrics, and CoPP policies
  • Troubleshoot network packet switching and forwarding
  • Properly design, implement, and troubleshoot issues related to Virtual Port Channels (VPC and VPC+)
  • Optimize routing through filtering or path manipulation
  • Optimize IP/IPv6 services and FHRP protocols (including HSRP, VRRP, and Anycast HSRP)
  • Troubleshoot EIGRP, OSPF, and IS-IS neighbor relationships and routing paths
  • Identify and resolve issues with Nexus route maps
  • Locate problems with BGP neighbor adjacencies and enhance path selection
  • Troubleshoot high availability components (BFD, SSO, ISSU, and GIR)
  • Understand multicast protocols and troubleshooting techniques
  • Identify and solve problems with OTV
  • Use NX-OS APIs to automate troubleshooting and administrative tasks

Table of contents

  1. Cover
  2. Title Page
  3. Copyright Page
  4. About the Authors
  5. About the Technical Reviewers
  6. Dedication
  7. Acknowledgments
  8. Contents at a Glance
  9. Reader Services
  10. Contents
  11. Icons Used in This Book
  12. Command Syntax Conventions
  13. Foreword
  14. Introduction
  15. Part I Introduction to Troubleshooting Nexus Switches
    1. Chapter 1 Introduction to Nexus Operating System (NX-OS)
      1. Nexus Platforms Overview
        1. Nexus 2000 Series
        2. Nexus 3000 Series
        3. Nexus 5000 Series
        4. Nexus 6000 Series
        5. Nexus 7000 Series
        6. Nexus 9000 Series
      2. NX-OS Architecture
        1. The Kernel
        2. System Manager (sysmgr)
        3. Messages and Transactional Services
        4. Persistent Storage Services
        5. Feature Manager
        6. NX-OS Line Card Microcode
        7. File Systems
          1. Flash File System
          2. Onboard Failure Logging
          3. Logflash
      3. Understanding NX-OS Software Releases and Packaging
        1. Software Maintenance Upgrades
        2. Licensing
      4. NX-OS High-Availability Infrastructure
        1. Supervisor Redundancy
        2. ISSU
      5. NX-OS Virtualization Features
        1. Virtual Device Contexts
        2. Virtual Routing and Forwarding
        3. Virtual Port Channel
      6. Management and Operations Capabilities
        1. NX-OS Advanced CLI
        2. Technical Support Files
        3. Accounting Log
        4. Feature Event-History
        5. Debug Options: Log File and Filters
        6. Configuration Checkpoint and Rollback
        7. Consistency Checkers
        8. Feature Scheduler, EEM, and Python
        9. Bash Shell
      7. Summary
      8. References
    2. Chapter 2 NX-OS Troubleshooting Tools
      1. Packet Capture: Network Sniffer
        1. Encapsulated Remote SPAN
        2. SPAN on Latency and Drop
          1. SPAN-on-Latency
          2. SPAN-on-Drop
      2. Nexus Platform Tools
        1. Ethanalyzer
        2. Packet Tracer
      3. NetFlow
        1. NetFlow Configuration
          1. Enable NetFlow Feature
          2. Define a Flow Record
          3. Define a Flow Exporter
          4. Define and Apply the Flow Monitor
        2. NetFlow Sampling
        3. sFlow
      4. Network Time Protocol
      5. Embedded Event Manager
      6. Logging
        1. Debug Logfiles
        2. Accounting Log
        3. Event-History
      7. Summary
      8. References
    3. Chapter 3 Troubleshooting Nexus Platform Issues
      1. Troubleshooting Hardware Issues
        1. Generic Online Diagnostic Tests
          1. Bootup Diagnostics
          2. Runtime Diagnostics
          3. GOLD Test and EEM Support
        2. Nexus Device Health Checks
          1. Hardware and Process Crashes
          2. Packet Loss
          3. Interface Errors and Drops
          4. Platform-Specific Drops
        3. Nexus Fabric Extenders
      2. Virtual Device Context
        1. VDC Resource Template
        2. Configuring VDC
        3. VDC Initialization
        4. Out-of-Band and In-Band Management
        5. VDC Management
          1. Line Card Interop Limitations
      3. Troubleshooting NX-OS System Components
        1. Message and Transaction Services
        2. Netstack and Packet Manager
          1. Netstack TCPUDP Component
        3. ARP and Adjacency Manager
          1. Unicast Forwarding Components
          2. Unicast Routing Information Base
          3. UFDM and IPFIB
        4. EthPM and Port-Client
      4. HWRL, CoPP, and System QoS
        1. MTU Settings
          1. FEX Jumbo MTU Settings
          2. Troubleshooting MTU Issues
      5. Summary
      6. References
  16. Part II Troubleshooting Layer 2 Forwarding
    1. Chapter 4 Nexus Switching
      1. Network Layer 2 Communication Overview
      2. Virtual LANs
        1. VLAN Creation
        2. Access Ports
        3. Trunk Ports
          1. Native VLANs
          2. Allowed VLANs
        4. Private VLANS
          1. Isolated Private VLANs
          2. Community Private VLANs
          3. Using a Promiscuous PVLAN Port on Switched Virtual Interface
          4. Trunking PVLANs Between Switches
      3. Spanning Tree Protocol Fundamentals
        1. IEEE 802.1D Spanning Tree Protocol
        2. Rapid Spanning Tree Protocol
          1. Spanning-Tree Path Cost
          2. Root Bridge Election
          3. Locating Root Ports
          4. Locating Blocked Switch Ports
          5. Verification of VLANS on Trunk Links
          6. Spanning Tree Protocol Tuning
        3. Multiple Spanning-Tree Protocol (MST)
          1. MST Configuration
          2. MST Verification
          3. MST Tuning
      4. Detecting and Remediating Forwarding Loops
        1. MAC Address Notifications
        2. BPDU Guard
        3. BPDU Filter
        4. Problems with Unidirectional Links
          1. Spanning Tree Protocol Loop Guard
          2. Unidirectional Link Detection
          3. Bridge Assurance
      5. Summary
      6. References
    2. Chapter 5 Port-Channels, Virtual Port-Channels, and FabricPath
      1. Port-Channels
        1. Basic Port-Channel Configuration
        2. Verifying Port-Channel Status
        3. Verifying LACP Packets
        4. Advanced LACP Configuration Options
          1. Minimum Number of Port-Channel Member Interfaces
          2. Maximum Number of Port-Channel Member Interfaces
        5. LACP System Priority
          1. LACP Interface Priority
          2. LACP Fast
          3. Graceful Convergence
          4. Suspend Individual
        6. Port-Channel Member Interface Consistency
        7. Troubleshooting LACP Interface Establishment
        8. Troubleshooting Traffic Load-Balancing
      2. Virtual Port-Channel
        1. vPC Fundamentals
          1. vPC Domain
          2. vPC Peer-Keepalive
          3. vPC Peer Link
          4. vPC Member Links
          5. vPC Operational Behavior
        2. vPC Configuration
        3. vPC Verification
          1. Verifying the vPC Domain Status
          2. Verifying the Peer-Keepalive
          3. vPC Consistency-Checker
        4. Advanced vPC Features
          1. vPC Orphan Ports
          2. vPC Autorecovery
          3. vPC Peer-Gateway
          4. vPC ARP Synchronization
          5. Backup Layer 3 Routing
          6. Layer 3 Routing over vPC
      3. FabricPath
        1. FabricPath Terminologies and Components
        2. FabricPath Packet Flow
        3. FabricPath Configuration
        4. FabricPath Verification and Troubleshooting
        5. FabricPath Devices
      4. Emulated Switch and vPC+
        1. vPC+ Configuration
        2. vPC+ Verification and Troubleshooting
      5. Summary
      6. References
  17. Part III Troubleshooting Layer 3 Routing
    1. Chapter 6 Troubleshooting IP and IPv6 Services
      1. IP SLA
        1. ICMP Echo Probe
        2. UDP Echo Probe
        3. UDP Jitter Probe
        4. TCP Connect Probe
      2. Object Tracking
        1. Object Tracking for the Interface
        2. Object Tracking for Route State
        3. Object Tracking for Track-List State
        4. Using Track Objects with Static Routes
      3. IPv4 Services
        1. DHCP Relay
        2. DHCP Snooping
        3. Dynamic ARP Inspection
          1. ARP ACLs
        4. IP Source Guard
        5. Unicast RPF
      4. IPv6 Services
        1. Neighbor Discovery
        2. IPv6 Address Assignment
          1. DHCPv6 Relay Agent
          2. DHCPv6 Relay LDRA
        3. IPv6 First-Hop Security
          1. RA Guard
          2. IPv6 Snooping
          3. DHCPv6 Guard
      5. First-Hop Redundancy Protocol
        1. HSRP
          1. HSRPv6
        2. VRRP
        3. GLBP
      6. Summary
    2. Chapter 7 Troubleshooting Enhanced Interior Gateway Routing Protocol (EIGRP)
      1. EIGRP Fundamentals
        1. Topology Table
        2. Path Metric Calculation
        3. EIGRP Communication
        4. Baseline EIGRP Configuration
      2. Troubleshooting EIGRP Neighbor Adjacency
        1. Verification of Active Interfaces
        2. Passive Interface
        3. Verification of EIGRP Packets
        4. Connectivity Must Exist Using the Primary Subnet
        5. EIGRP ASN Mismatch
        6. Mismatch K Values
        7. Problems with Hello and Hold Timers
        8. EIGRP Authentication Issues
          1. Interface-Based EIGRP Authentication
          2. Global EIGRP Authentication
      3. Troubleshooting Path Selection and Missing Routes
        1. Load Balancing
        2. Stub
        3. Maximum-Hops
        4. Distribute List
        5. Offset Lists
        6. Interface-Based Settings
        7. Redistribution
        8. Classic Metrics vs. Wide Metrics
      4. Problems with Convergence
        1. Active Query
        2. Stuck in Active
      5. Summary
      6. References
    3. Chapter 8 Troubleshooting Open Shortest Path First (OSPF)
      1. OSPF Fundamentals
        1. Inter-Router Communication
        2. OSPF Hello Packets
        3. Neighbor States
        4. Designated Routers
        5. Areas
        6. Link State Advertisements
      2. Troubleshooting OSPF Neighbor Adjacency
        1. Baseline OSPF Configuration
        2. OSPF Neighbor Verification
        3. Confirmation of OSPF Interfaces
        4. Passive Interface
        5. Verification of OSPF Packets
        6. Connectivity Must Exist Using the Primary Subnet
        7. MTU Requirements
        8. Unique Router-ID
        9. Interface Area Numbers Must Match
        10. OSPF Stub (Area Flags) Settings Must Match
        11. DR Requirements
        12. Timers
        13. Authentication
      3. Troubleshooting Missing Routes
        1. Discontiguous Network
        2. Duplicate Router ID
        3. Filtering Routes
        4. Redistribution
        5. OSPF Forwarding Address
      4. Troubleshooting OSPF Path Selection
        1. Intra-Area Routes
        2. Inter-Area Routes
        3. External Route Selection
        4. E1 and N1 External Routes
        5. E2 and N2 External Routes
        6. Problems with Intermixed RFC 1583 and RFC 2328 Devices
        7. Interface Link Costs
      5. Summary
      6. References
    4. Chapter 9 Troubleshooting Intermediate System-Intermediate System (IS-IS)
      1. IS-IS Fundamentals
        1. Areas
        2. NET Addressing
        3. Inter-Router Communication
        4. IS Protocol Header
        5. TLVs
        6. IS PDU Addressing
        7. IS-IS Hello (IIH) Packets
        8. Link-State Packets
          1. LSP ID
          2. Attribute Fields
          3. LSP Packet and TLVs
        9. Designated Intermediate System
        10. Path Selection
      2. Troubleshooting IS-IS Neighbor Adjacency
        1. Baseline IS-IS Configuration
        2. IS-IS Neighbor Verification
        3. Confirmation of IS-IS Interfaces
        4. Passive Interface
        5. Verification of IS-IS Packets
        6. Connectivity Must Exist Using the Primary Subnet
        7. MTU Requirements
        8. Unique System-ID
        9. Area Must Match Between L1 Adjacencies
        10. Checking IS-IS Adjacency Capabilities
        11. DIS Requirements
        12. IIH Authentication
      3. Troubleshooting Missing Routes
        1. Duplicate System ID
        2. Interface Link Costs
        3. Mismatch of Metric Modes
        4. L1 to L2 Route Propagations
        5. Suboptimal Routing
        6. Redistribution
      4. Summary
      5. References
    5. Chapter 10 Troubleshooting Nexus Route-Maps
      1. Conditional Matching
        1. Access Control Lists
        2. ACLs and ACL Manager Component
          1. Interior Gateway Protocol (IGP) Network Selection
          2. BGP Network Selection
        3. Prefix Matching and Prefix-Lists
          1. Prefix Matching
          2. Prefix Lists
      2. Route-Maps
        1. Conditional Matching
          1. Multiple Conditional Match Conditions
          2. Complex Matching
        2. Optional Actions
        3. Incomplete Configuration of Routing Policies
        4. Diagnosing Route Policy Manger
      3. Policy-Based Routing
      4. Summary
      5. References
    6. Chapter 11 Troubleshooting BGP
      1. BGP Fundamentals
        1. Address Families
        2. Path Attributes
        3. Loop Prevention
      2. BGP Sessions
        1. BGP Identifier
        2. BGP Messages
          1. OPEN
          2. UPDATE
          3. NOTIFICATION
          4. KEEPALIVE
        3. BGP Neighbor States
          1. Idle
          2. Connect
          3. Active
          4. OpenSent
          5. OpenConfirm
          6. Established
        4. BGP Configuration and Verification
      3. Troubleshooting BGP Peering Issues
        1. Troubleshooting BGP Peering Down Issues
          1. Verifying Configuration
          2. Verifying Reachability and Packet Loss
          3. Verifying ACLs and Firewalls in the Path
          4. Verifying TCP Sessions
          5. OPEN Message Errors
          6. BGP Debugs
        2. Demystifying BGP Notifications
        3. Troubleshooting IPv6 Peers
        4. BGP Peer Flapping Issues
          1. Bad BGP Update
          2. Hold Timer Expired
          3. BGP Keepalive Generation
          4. MTU Mismatch Issues
      4. BGP Route Processing and Route Propagation
        1. BGP Route Advertisement
          1. Network Statement
          2. Redistribution
          3. Route Aggregation
          4. Default-Information Originate
        2. BGP Best Path Calculation
        3. BGP Multipath
          1. EBGP and IBGP Multipath
        4. BGP Update Generation Process
        5. BGP Convergence
      5. Scaling BGP
        1. Tuning BGP Memory
          1. Prefixes
          2. Paths
          3. Attributes
          4. Scaling BGP Configuration
        2. Soft Reconfiguration Inbound Versus Route Refresh
        3. Scaling BGP with Route-Reflectors
          1. Loop Prevention in Route Reflectors
        4. Maximum Prefixes
        5. BGP Max AS
      6. BGP Route Filtering and Route Policies
        1. Prefix-List-Based Filtering
        2. Filter-Lists
      7. BGP Route-Maps
        1. Regular Expressions (RegEx)
          1. _ Underscore
          2. ^ Caret
          3. $ Dollar Sign
          4. [ ] Brackets
          5. - Hyphen
          6. [^] Caret in Brackets
          7. ( ) Parentheses and | Pipe
          8. . Period
          9. + Plus Sign
          10. ? Question Mark
          11. * Asterisk
        2. AS-Path Access List
        3. BGP Communities
      8. Looking Glass and Route Servers
      9. Logs Collection
      10. Summary
      11. Further Reading
      12. References
  18. Part IV Troubleshooting High Availability
    1. Chapter 12 High Availability
      1. Bidirectional Forwarding Detection
        1. Asynchronous Mode
        2. Asynchronous Mode with Echo Function
        3. Configuring and Verifying BFD Sessions
      2. Nexus High Availability
        1. Stateful Switchover
        2. ISSU
      3. Graceful Insertion and Removal
        1. Custom Maintenance Profile
      4. Summary
      5. References
  19. Part V Multicast Network Traffic
    1. Chapter 13 Troubleshooting Multicast
      1. Multicast Fundamentals
        1. Multicast Terminology
        2. Layer 2 Multicast Addresses
        3. Layer 3 Multicast Addresses
      2. NX-OS Multicast Architecture
        1. Replication
        2. Protecting the Central Processing Unit
        3. NX-OS Multicast Implementation
          1. Static Joins
          2. Clearing an MROUTE Entry
          3. Multicast Boundary and Filtering
          4. Event-Histories and Show Techs
      3. IGMP
        1. IGMPv2
        2. IGMPv3
        3. IGMP Snooping
        4. IGMP Verification
      4. PIM Multicast
        1. PIM Protocol State and Trees
        2. PIM Message Types
          1. PIM Hello Message
          2. PIM Register Message
          3. PIM Register-Stop Message
          4. PIM Join-Prune Message
          5. PIM Bootstrap Message
          6. PIM Assert Message
          7. PIM Candidate RP Advertisement Message
          8. PIM DF Election Message
        3. PIM Interface and Neighbor Verification
        4. PIM Any Source Multicast
          1. PIM ASM Configuration
          2. PIM ASM Verification
          3. PIM ASM Event-History and MROUTE State Verification
          4. PIM ASM Platform Verification
        5. PIM Bidirectional
          1. BiDIR Configuration
          2. BiDIR Verification
        6. PIM RP Configuration
          1. Static RP Configuration
          2. Auto-RP Configuration and Verification
          3. BSR Configuration and Verification
          4. Anycast-RP Configuration and Verification
          5. Anycast RP with MSDP
          6. PIM Anycast RP
        7. PIM Source Specific Multicast
          1. SSM Configuration
          2. SSM Verification
      5. Multicast and Virtual Port-Channel
        1. vPC-Connected Source
        2. vPC-Connected Receiver
        3. vPC Considerations for Multicast Traffic
          1. Duplicate Multicast Packets
          2. Reserved VLAN
      6. Ethanalyzer Examples
      7. Summary
      8. References
  20. Part VI Troubleshooting Nexus Tunneling
    1. Chapter 14 Troubleshooting Overlay Transport Virtualization (OTV)
      1. OTV Fundamentals
        1. Flood Control and Broadcast Optimization
        2. Supported OTV Platforms
        3. OTV Terminology
        4. Deploying OTV
          1. OTV Deployment Models
          2. OTV Site VLAN
          3. OTV Configuration
      2. Understanding and Verifying the OTV Control Plane
        1. OTV Multicast Mode
        2. OTV IS-IS Adjacency Verification
        3. OTV IS-IS Topology Table
        4. OTV IS-IS Authentication
        5. Adjacency Server Mode
        6. OTV Control Plane Policing (CoPP)
      3. Understanding and Verifying the OTV Data Plane
        1. OTV ARP Resolution and ARP-ND-Cache
        2. Broadcasts
        3. Unknown Unicast Frames
        4. OTV Unicast Traffic with a Multicast Enabled Transport
        5. OTV Multicast Traffic with a Multicast Enabled Transport
        6. OTV Multicast Traffic with a Unicast Transport (Adjacency Server Mode)
      4. Advanced OTV Features
        1. First Hop Routing Protocol Localization
        2. Multihoming
        3. Ingress Routing Optimization
        4. VLAN Translation
        5. OTV Tunnel Depolarization
        6. OTV Fast Failure Detection
      5. Summary
      6. References
  21. Part VII Network Programmability
    1. Chapter 15 Programmability and Automation
      1. Introduction to Automation and Programmability
      2. Introduction to Open NX-OS
        1. Shells and Scripting
          1. Bash Shell
          2. Guest Shell
          3. Python
      3. NX-SDK
      4. NX-API
      5. Summary
      6. References
  22. Index

Product information

  • Title: Troubleshooting Cisco Nexus Switches and NX-OS, First Edition
  • Author(s): Vinit Jain, Brad Edgeworth, Richard Furr
  • Release date: June 2018
  • Publisher(s): Cisco Press
  • ISBN: 9780134783208