Inside Windows® Debugging

Inside Windows® Debugging

by Tarik Soulami
Released May 2012
Publisher(s): Microsoft Press
ISBN: 9780735671348

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Use Windows debuggers throughout the development cycle—and build better software

Rethink your use of Windows debugging and tracing tools—and learn how to make them a key part of test-driven software development. Led by a member of the Windows Fundamentals Team at Microsoft, you’ll apply expert debugging and tracing techniques—and sharpen your C++ and C# code analysis skills—through practical examples and common scenarios. Learn why experienced developers use debuggers in every step of the development process, and not just when bugs appear.

Discover how to:

  • Go behind the scenes to examine how powerful Windows debuggers work

  • Catch bugs early in the development cycle with static and runtime analysis tools

  • Gain practical strategies to tackle the most common code defects

  • Apply expert tricks to handle user-mode and kernel-mode debugging tasks

  • Implement postmortem techniques such as JIT and dump debugging

  • Debug the concurrency and security aspects of your software

  • Use debuggers to analyze interactions between your code and the operating system

  • Analyze software behavior with Xperf and the Event Tracing for Windows (ETW) framework

    •

    Table of contents

    1. Inside Windows® Debugging
    2. Table of Contents
    3. Foreword
    4. Introduction
      1. Who Should Read This Book
        1. Assumptions
      2. Organization of This Book
      3. Conventions in This Book
      4. System Requirements
      5. Code Samples
        1. Installing the Code Samples
        2. Running the Code Samples
          1. Compiling the Code Samples
          2. Compiling the .NET Code Samples
          3. Compiling the C/C++ Code Samples
      6. Acknowledgments
      7. Errata & Book Support
      8. We Want to Hear from You
      9. Stay in Touch
    5. 1. A Bit of Background
      1. 1. Software Development in Windows
        1. Windows Evolution
          1. Windows Release History
          2. Supported CPU Architectures
          3. Windows Build Flavors
          4. Windows Servicing Terminology
        2. Windows Architecture
          1. Kernel Mode vs. User Mode
          2. User-Mode System Processes
          3. User-Mode Application Processes
          4. Low-Level Windows Communication Mechanisms
            1. Calling Kernel-Mode Code from User Mode
            2. Calling User-Mode Code from Kernel Mode
            3. Interprocess Communication
        3. Windows Developer Interface
          1. Developer Documentation Resources
          2. WDM, KMDF, and UMDF
          3. The NTDLL and USER32 Layers
          4. The Win32 API Layer
          5. The COM Layer
            1. COM in the Windows Operating System
            2. Writing COM Servers
            3. Consuming COM Objects
          6. The CLR (.NET) Layer
            1. .NET Side-by-Side Versioning
            2. .NET Executable Programs Load Sequence
            3. Interoperability Between Managed and Native Code
        4. Microsoft Developer Tools
          1. The Windows DDK (WDK)
          2. The Windows SDK
        5. Summary
    6. 2. Debugging for Fun and Profit
      1. 2. Getting Started
        1. Introducing the Debugging Tools
          1. Acquiring the Windows Debuggers Package
            1. Installing the Windows Debuggers
              1. Installing the Windows Debuggers
          2. Acquiring the Visual Studio Debugger
            1. Installing Visual Studio 2010 Ultimate (Trial Version)
              1. Installing Visual Studio 2010 Ultimate (Trial Version)
          3. Comparing the WinDbg and Visual Studio Debuggers
        2. User-Mode Debugging
          1. Debugging Your First Program with WinDbg
            1. Debugging Your Own Code
            2. Debugging System Code
          2. Listing the Values of Local Variables and Function Parameters
            1. Listing Parameters and Locals for Your Own Code
            2. Listing Parameters and Locals for System Code
          3. Source-Level Debugging in WinDbg
          4. Symbol Files, Servers, and Local Caches
            1. Public vs. Private Symbols
            2. Microsoft Symbols Server
          5. Caching Symbols Offline for WinDbg
          6. Troubleshooting Symbol Resolution Issues in WinDbg
          7. Name Decoration Considerations
          8. Getting Help for WinDbg Commands
        3. Kernel-Mode Debugging
          1. Your First (Live) Kernel Debugging Session
            1. Enabling Kernel-Mode Debugging for a (Target) Machine
              1. Enabling Kernel-Mode Debugging for a (Target) Machine
              2. Starting a Live Kernel-Mode Debugging Session
          2. Setting Up a Kernel-Mode Debugging Environment Using Physical Machines
            1. KD Cabling Options
            2. Step-by-Step Guide to Enabling KD over 1394 Ports
              1. Starting a Kernel-Mode Debugging Session over 1394
          3. Setting Up a Kernel-Mode Debugging Environment Using Virtual Machines
            1. The Microsoft Hypervisor Technology
            2. Configuring Kernel Debugging over a Virtualized COM Port
              1. Starting a Kernel-Mode Debugging Session over a Virtualized COM Port
            3. Benefits of Testing Using Virtual Machines
          4. Diagnosing Host/Target Communication Issues
          5. Understanding the KD Break-in Sequence
          6. Controlling the Target in the Kernel Debugger
          7. Setting Code Breakpoints in the Kernel Debugger
            1. Setting Code Breakpoints in User-Mode Memory
            2. Setting Code Breakpoints in Kernel-Mode Memory
          8. Getting Help for WinDbg Kernel Debugging Commands
        4. Summary
      2. 3. How Windows Debuggers Work
        1. User-Mode Debugging
          1. Architecture Overview
          2. Win32 Debugging APIs
          3. Debug Events and Exceptions
            1. .NET, C++, and SEH Exceptions
            2. SEH Exception Handling in the User-Mode Debugger
          4. The Break-in Sequence
          5. Setting Code Breakpoints
          6. Observing Code Breakpoint Insertion in WinDbg
        2. Kernel-Mode Debugging
          1. Architecture Overview
          2. Setting Code Breakpoints
          3. Single-Stepping the Target
          4. Switching the Current Process Context
        3. Managed-Code Debugging
          1. Architecture Overview
          2. The SOS Windows Debuggers Extension
            1. How SOS Works
            2. Debugging Your First .NET Program Using SOS
        4. Script Debugging
          1. Architecture Overview
          2. Debugging Scripts in Visual Studio
        5. Remote Debugging
          1. Architecture Overview
          2. Remote Debugging in WinDbg
            1. Remote Sessions
              1. Using Remote Sessions in WinDbg
            2. Remote Stubs
              1. Using Remote Stubs in WinDbg
          3. Remote Debugging in Visual Studio
            1. Using Remote Debugging in Visual Studio
              1. Using Remote Debugging in Visual Studio
        6. Summary
      3. 4. Postmortem Debugging
        1. Just-in-Time Debugging
          1. Your First JIT Debugging Experiment
          2. How Just-in-Time Debugging Works
          3. Using Visual Studio as Your JIT Debugger
            1. Native JIT Debugging
            2. Managed Code (.NET) JIT Debugging
            3. Script JIT Debugging
          4. Run-Time Assertions and JIT Debugging
          5. JIT Debugging in Session 0
        2. Dump Debugging
          1. Automatic User-Mode, Crash-Dump Generation
            1. The LocalDumps Registry Key
            2. Your First Dump Debugging Experiment
          2. Analyzing Crash Dumps Using the WinDbg Debugger
            1. Finding SEH Crash Exceptions
            2. Finding .NET Crash Exceptions
            3. Moving Beyond the Immediate Symptom
          3. Analyzing Crash Dumps in Visual Studio
          4. Manual Dump-File Generation
          5. “Time Travel” Debugging
          6. Kernel-Mode Postmortem Debugging
        3. Summary
      4. 5. Beyond the Basics
        1. Noninvasive Debugging
        2. Data Breakpoints
          1. Deep Inside User-Mode and Kernel-Mode Data Breakpoints
          2. Clearing Kernel-Mode Data Breakpoints
          3. Execution Data Breakpoints vs. Code Breakpoints
          4. User-Mode Debugger Data Breakpoints in Action: C++ Global Objects and the C Runtime Library
          5. Kernel-Mode Debugger Data Breakpoints in Action: Waiting for a Process to Exit
          6. Advanced Example: Who Is Changing a Registry Value?
        3. Scripting the Debugger
          1. Replaying Commands Using Debugger Scripts
          2. Debugger Pseudo-Registers
          3. Resolving C++ Template Names in Debugger Scripts
          4. Scripts in Action: Listing Windows Service Processes in the Kernel Debugger
        4. WOW64 Debugging
          1. The WOW64 Environment
          2. Debugging of WOW64 Processes
        5. Windows Debugging Hooks (GFLAGS)
          1. Systemwide vs. Process-Specific NT Global Flags
          2. The GFLAGS Tool
          3. The !gflag Debugger Extension Command
          4. Impact of User-Mode Debuggers on the Value of the NT Global Flag
          5. The Image File Execution Options Hooks
        6. Summary
      5. 6. Code Analysis Tools
        1. Static Code Analysis
          1. Catching Your First Crashing Bug Using VC++ Static Code Analysis
          2. SAL Annotations
            1. Catching a Buffer Overrun Bug Using SAL Annotations
            2. SAL in the Win32 API Headers
          3. Other Static Analysis Tools
            1. OACR in the DDK Build Environment
            2. Standalone FxCop Tool
        2. Runtime Code Analysis
          1. Catching Your First Bug Using the Application Verifier Tool
          2. A Behind-the-Scenes Look: Verifier Support in the Operating System
          3. The !avrf Debugger Extension Command
          4. The Application Verifier as a Quality Assurance Tool
        3. Summary
      6. 7. Expert Debugging Tricks
        1. Essential Tricks
          1. Waiting for a Debugger to Attach to the Target
          2. Breaking on DLL Load Events
            1. Breaking on DLL Load Events in the WinDbg Debugger
            2. Breaking on DLL Load Events in the Visual Studio Debugger
          3. Debugging Process Startup
            1. Startup Debugging in Action
            2. How the Startup Debugger Hook Works
            3. Startup Debugging of Service Processes in Session 0
          4. Debugging Child Processes
            1. Child Debugging in Action
            2. How Child Debugging Works
            3. Child Debugging as a “Startup Debugger” Alternative
            4. Example: The Case of MSI Custom Actions
        2. More Useful Tricks
          1. Debugging Error-Code Failures
            1. Debugging Win32 API Failures
            2. Observing Low-Level Resource Access Failures with the Process Monitor Tool
            3. Debugging High-Level, Error-Code Failures
            4. Tracking Down Error Codes the Brute-Force Way
          2. Breaking on First-Chance Exception Notifications
          3. Freezing Threads
            1. Freezing and Unfreezing Threads
            2. The Difference Between Freezing and Suspending Threads
        3. Kernel-Mode Debugging Tricks
          1. Breaking on User-Mode Process Creation
            1. The nt!PspInsertProcess Kernel-Mode Breakpoint
            2. Breaking on Instances of a Particular Process Only
            3. Tracing Processes Started During the Windows Boot Sequence
          2. Debugging the Startup of User-Mode Processes
          3. Breaking on DLL Load Events
          4. Breaking on Unhandled SEH Exceptions
          5. Freezing Threads
        4. Summary
      7. 8. Common Debugging Scenarios, Part 1
        1. Debugging Access Violations
          1. Understanding Memory Access Violations
          2. The !analyze Debugger Extension Command
        2. Debugging Heap Corruptions
          1. Debugging Native Heap Corruptions
            1. Introducing the Page Heap
            2. How Page Heap Works
            3. Enabling Page Heap for a Target Process
            4. The !heap Debugger Extension Command
            5. Customizing Page Heap for Your Situation
          2. Debugging Managed (GC) Heap Corruptions
            1. The !VerifyHeap SOS Command
            2. Managed Debugging Assistants
        3. Debugging Stack Corruptions
          1. Stack-Based Buffer Overruns
          2. Using Data Breakpoints in Stack Corruption Investigations
          3. Reconstructing Call Frames from Corrupted Stacks
        4. Debugging Stack Overflows
          1. Understanding Stack Overflows
          2. The kf Debugger Command
        5. Debugging Handle Leaks
          1. A Handle Leak Example
          2. The !htrace Debugger Extension Command
            1. How !htrace Works
            2. !htrace in Action
            3. Limitations of Handle-Leak Detection Using !htrace
        6. Debugging User-Mode Memory Leaks
          1. Detecting Resource Leaks Using the Application Verifier Tool
          2. Investigating Memory Leaks Using the UMDH Tool
            1. How UMDH Works
            2. UMDH in Action
            3. Resolving Symbols in UMDH
          3. Extending the Strategy: A Custom Reference Stack-Trace Database
        7. Debugging Kernel-Mode Memory Leaks
          1. Kernel Memory Basics
          2. Investigating Kernel-Mode Leaks Using Pool Tagging
        8. Summary
      8. 9. Common Debugging Scenarios, Part 2
        1. Debugging Race Conditions
          1. Shared-State Consistency Bugs
            1. Stress Testing and Reproducing Race Conditions
            2. Simulating Race Conditions Under the Debugger
            3. Shared-State Consistency Bugs: A Few Suggested Fixes
          2. Shared-State Lifetime Management Bugs
            1. Thread Context Lifetime and Reference Counting
            2. Global Variables in Worker Threads and the Process Rundown Sequence
          3. DLL Module Lifetime-Management Bugs
            1. Debugging Unloaded Code
            2. DLL Module Lifetime-Management Bugs: A Few Suggested Fixes
        2. Debugging Deadlocks
          1. Lock-Ordering Deadlocks
            1. Debugging the Deadlock Condition
            2. The !cs Debugger Extension Command
            3. Avoiding Lock-Ordering Deadlocks
          2. Logical Deadlocks
            1. Debugging the Deadlock Condition
            2. The Loader Lock and DllMain Hazards
            3. The Loader Lock and the Debugger Break-in Sequence
        3. Debugging Access-Check Problems
          1. The Basic NT Security Model
            1. Anatomy of an Access Check
            2. The !token Debugger Extension Command
            3. The !sd Debugger Extension Command
          2. Windows Vista Improvements
            1. Integrity Levels
            2. UAC in Action
          3. Wrapping Up
        4. Summary
      9. 10. Debugging System Internals
        1. The Windows Console Subsystem
          1. The Magic Behind printf
            1. The Console Host Process in Windows 7
            2. The C Runtime Library’s I/O Functions
          2. Handling of Windows UI Events
          3. Handling of the Ctrl+C Signal
            1. Console Termination Handlers in .NET/Win32
            2. Console Termination Handlers at the System Level
        2. Anatomy of System Calls
          1. The User-Mode Side of System Calls
          2. The Transition into Kernel Mode
          3. The Kernel-Mode Side of System Calls
        3. Summary
    7. 3. Observing and Analyzing Software Behavior
      1. 11. Introducing Xperf
        1. Acquiring Xperf
          1. Installing the Windows Performance Toolkit
            1. Installing the Windows Performance Toolkit
              1. Installing the Windows Performance Toolkit
        2. Your First Xperf Investigation
          1. Devising an Investigation Strategy
          2. Collecting an ETW Trace for the Scenario
          3. Analyzing the Collected ETW Trace
            1. Phase 1: Narrowing the List of Bottlenecks
            2. Phase 2: Getting Stack Traces for the Hot Spots
        3. Xperf’s Strengths and Limitations
        4. Summary
      2. 12. Inside ETW
        1. ETW Architecture
          1. ETW Design Principles
          2. ETW Components
          3. The Special NT Kernel Logger Session
          4. Configuring ETW Sessions Using Xperf
            1. Filled Buffers and Lost ETW Events
            2. Using Circular ETW Logging
        2. Existing ETW Instrumentation in Windows
          1. Instrumentation in the Windows Kernel
            1. The PROC_THREAD and LOADER Kernel Flags
            2. Other Kernel Flags and Groups
            3. Viewing Kernel Provider Events in Xperf
          2. Instrumentation in Other Windows Components
            1. Manifest-Based User Providers
            2. Discovering User Providers
            3. Viewing User Provider Events in Xperf
        3. Understanding ETW Stack-Walk Events
          1. Enabling and Viewing Stack Traces for Kernel Provider Events
          2. Enabling and Viewing Stack Traces for User Provider Events
          3. Diagnosing ETW Stack-Trace Issues
            1. Diagnosing Missing Call Stacks
            2. Diagnosing Unresolved Call Stacks
            3. Diagnosing Incomplete Call Stacks
        4. Adding ETW Logging to Your Code
          1. Anatomy of ETW Events
          2. Logging Events Using the ETW Win32 APIs
        5. Boot Tracing in ETW
          1. Logging Kernel Provider Events During Boot
          2. Logging User Provider Events During Boot
        6. Summary
      3. 13. Common Tracing Scenarios
        1. Analyzing Blocked Time
          1. The CSwitch and ReadyThread ETW Events
          2. Wait Analysis Using Visual Studio 2010
            1. Starting a Parallel Performance Analyzer Tracing Session
              1. Starting a Parallel Performance Analyzer Tracing Session
          3. Wait Analysis Using Xperf
        2. Analyzing Memory Usage
          1. Analyzing High-Level Memory Usage in a Target Process
          2. Analyzing NT Heap Memory Usage
            1. Capturing ETW Heap Traces Using Xperf
            2. Analyzing ETW Heap Traces Using Xperf
          3. Analyzing GC Heap (.NET) Memory Usage
            1. Analyzing GC Heap Memory Using PerfView
              1. Collecting ETW Traces Using PerfView
            2. Analyzing GC Heap Memory Using SOS
        3. Tracing as a Debugging Aid
          1. Tracing Error Code Failures
            1. Why Trace When You Can Debug?
            2. Tracing System Call Failures
          2. Tracing System Internals
            1. Devising an Exploration Strategy
            2. Observing the Scenario Using Kernel ETW Events
            3. Observing the Scenario Using User ETW Events
        4. Summary
      4. A. WinDbg User-Mode Debugging Quick Start
        1. Starting a User-Mode Debugging Session
        2. Fixing the Symbols Path
        3. Fixing the Sources Path
        4. Displaying the Command Line of the Target Process
        5. Control Flow Commands
        6. Listing Loaded Modules and Their Version
        7. Resolving Function Addresses
        8. Setting Code (Software) Breakpoints
        9. Setting Data (Hardware) Breakpoints
        10. Switching Between Threads
        11. Displaying Call Stacks
        12. Displaying Function Parameters
        13. Displaying Local Variables
        14. Displaying Data Members of Native Types
        15. Navigating Between Call Frames
        16. Listing Function Disassembly
        17. Displaying and Modifying Memory and Register Values
        18. Ending a User-Mode Debugging Session
      5. B. WinDbg Kernel-Mode Debugging Quick Start
        1. Starting a Kernel-Mode Debugging Session
        2. Switching Between CPU Contexts
        3. Displaying Process Information
        4. Displaying Thread Information
        5. Switching Process and Thread Contexts
        6. Listing Loaded Modules and Their Version
        7. Setting Code (Software) Breakpoints Inside Kernel-Mode Code
        8. Setting Code (Software) Breakpoints Inside User-Mode Code
        9. Setting Data (Hardware) Breakpoints
        10. Ending a Kernel-Mode Debugging Session
    8. Index
    9. About the Author
    10. Copyright

    Product information

    • Title: Inside Windows® Debugging
    • Author(s): Tarik Soulami
    • Release date: May 2012
    • Publisher(s): Microsoft Press
    • ISBN: 9780735671348