book

Inside Windows® Debugging

Name: Inside Windows® Debugging
Author: Tarik Soulami
ISBN: 9780735671348

by Tarik Soulami

May 2012

Intermediate to advanced

592 pages

23h 52m

English

Microsoft Press

Read now

Unlock full access

Inside Windows® Debugging
Table of Contents
Foreword
Introduction
Who Should Read This BookAssumptionsOrganization of This BookConventions in This BookSystem RequirementsCode SamplesInstalling the Code SamplesRunning the Code SamplesCompiling the Code SamplesCompiling the .NET Code SamplesCompiling the C/C++ Code SamplesAcknowledgmentsErrata & Book SupportWe Want to Hear from YouStay in Touch
1. A Bit of Background
1. Software Development in Windows
Windows EvolutionWindows Release HistorySupported CPU ArchitecturesWindows Build FlavorsWindows Servicing TerminologyWindows ArchitectureKernel Mode vs. User ModeUser-Mode System ProcessesUser-Mode Application ProcessesLow-Level Windows Communication MechanismsCalling Kernel-Mode Code from User ModeCalling User-Mode Code from Kernel ModeInterprocess CommunicationWindows Developer InterfaceDeveloper Documentation ResourcesWDM, KMDF, and UMDFThe NTDLL and USER32 LayersThe Win32 API LayerThe COM LayerCOM in the Windows Operating SystemWriting COM ServersConsuming COM ObjectsThe CLR (.NET) Layer.NET Side-by-Side Versioning.NET Executable Programs Load SequenceInteroperability Between Managed and Native CodeMicrosoft Developer ToolsThe Windows DDK (WDK)The Windows SDKSummary
2. Debugging for Fun and Profit
2. Getting Started
Introducing the Debugging ToolsAcquiring the Windows Debuggers PackageInstalling the Windows DebuggersInstalling the Windows DebuggersAcquiring the Visual Studio DebuggerInstalling Visual Studio 2010 Ultimate (Trial Version)Installing Visual Studio 2010 Ultimate (Trial Version)Comparing the WinDbg and Visual Studio DebuggersUser-Mode DebuggingDebugging Your First Program with WinDbgDebugging Your Own CodeDebugging System CodeListing the Values of Local Variables and Function ParametersListing Parameters and Locals for Your Own CodeListing Parameters and Locals for System CodeSource-Level Debugging in WinDbgSymbol Files, Servers, and Local CachesPublic vs. Private SymbolsMicrosoft Symbols ServerCaching Symbols Offline for WinDbgTroubleshooting Symbol Resolution Issues in WinDbgName Decoration ConsiderationsGetting Help for WinDbg CommandsKernel-Mode DebuggingYour First (Live) Kernel Debugging SessionEnabling Kernel-Mode Debugging for a (Target) MachineEnabling Kernel-Mode Debugging for a (Target) MachineStarting a Live Kernel-Mode Debugging SessionSetting Up a Kernel-Mode Debugging Environment Using Physical MachinesKD Cabling OptionsStep-by-Step Guide to Enabling KD over 1394 PortsStarting a Kernel-Mode Debugging Session over 1394Setting Up a Kernel-Mode Debugging Environment Using Virtual MachinesThe Microsoft Hypervisor TechnologyConfiguring Kernel Debugging over a Virtualized COM PortStarting a Kernel-Mode Debugging Session over a Virtualized COM PortBenefits of Testing Using Virtual MachinesDiagnosing Host/Target Communication IssuesUnderstanding the KD Break-in SequenceControlling the Target in the Kernel DebuggerSetting Code Breakpoints in the Kernel DebuggerSetting Code Breakpoints in User-Mode MemorySetting Code Breakpoints in Kernel-Mode MemoryGetting Help for WinDbg Kernel Debugging CommandsSummary
3. How Windows Debuggers Work
User-Mode DebuggingArchitecture OverviewWin32 Debugging APIsDebug Events and Exceptions.NET, C++, and SEH ExceptionsSEH Exception Handling in the User-Mode DebuggerThe Break-in SequenceSetting Code BreakpointsObserving Code Breakpoint Insertion in WinDbgKernel-Mode DebuggingArchitecture OverviewSetting Code BreakpointsSingle-Stepping the TargetSwitching the Current Process ContextManaged-Code DebuggingArchitecture OverviewThe SOS Windows Debuggers ExtensionHow SOS WorksDebugging Your First .NET Program Using SOSScript DebuggingArchitecture OverviewDebugging Scripts in Visual StudioRemote DebuggingArchitecture OverviewRemote Debugging in WinDbgRemote SessionsUsing Remote Sessions in WinDbgRemote StubsUsing Remote Stubs in WinDbgRemote Debugging in Visual StudioUsing Remote Debugging in Visual StudioUsing Remote Debugging in Visual StudioSummary
4. Postmortem Debugging
Just-in-Time DebuggingYour First JIT Debugging ExperimentHow Just-in-Time Debugging WorksUsing Visual Studio as Your JIT DebuggerNative JIT DebuggingManaged Code (.NET) JIT DebuggingScript JIT DebuggingRun-Time Assertions and JIT DebuggingJIT Debugging in Session 0Dump DebuggingAutomatic User-Mode, Crash-Dump GenerationThe LocalDumps Registry KeyYour First Dump Debugging ExperimentAnalyzing Crash Dumps Using the WinDbg DebuggerFinding SEH Crash ExceptionsFinding .NET Crash ExceptionsMoving Beyond the Immediate SymptomAnalyzing Crash Dumps in Visual StudioManual Dump-File Generation“Time Travel” DebuggingKernel-Mode Postmortem DebuggingSummary

5. Beyond the Basics
Noninvasive DebuggingData BreakpointsDeep Inside User-Mode and Kernel-Mode Data BreakpointsClearing Kernel-Mode Data BreakpointsExecution Data Breakpoints vs. Code BreakpointsUser-Mode Debugger Data Breakpoints in Action: C++ Global Objects and the C Runtime LibraryKernel-Mode Debugger Data Breakpoints in Action: Waiting for a Process to ExitAdvanced Example: Who Is Changing a Registry Value?Scripting the DebuggerReplaying Commands Using Debugger ScriptsDebugger Pseudo-RegistersResolving C++ Template Names in Debugger ScriptsScripts in Action: Listing Windows Service Processes in the Kernel DebuggerWOW64 DebuggingThe WOW64 EnvironmentDebugging of WOW64 ProcessesWindows Debugging Hooks (GFLAGS)Systemwide vs. Process-Specific NT Global FlagsThe GFLAGS ToolThe !gflag Debugger Extension CommandImpact of User-Mode Debuggers on the Value of the NT Global FlagThe Image File Execution Options HooksSummary
6. Code Analysis Tools
Static Code AnalysisCatching Your First Crashing Bug Using VC++ Static Code AnalysisSAL AnnotationsCatching a Buffer Overrun Bug Using SAL AnnotationsSAL in the Win32 API HeadersOther Static Analysis ToolsOACR in the DDK Build EnvironmentStandalone FxCop ToolRuntime Code AnalysisCatching Your First Bug Using the Application Verifier ToolA Behind-the-Scenes Look: Verifier Support in the Operating SystemThe !avrf Debugger Extension CommandThe Application Verifier as a Quality Assurance ToolSummary
7. Expert Debugging Tricks
Essential TricksWaiting for a Debugger to Attach to the TargetBreaking on DLL Load EventsBreaking on DLL Load Events in the WinDbg DebuggerBreaking on DLL Load Events in the Visual Studio DebuggerDebugging Process StartupStartup Debugging in ActionHow the Startup Debugger Hook WorksStartup Debugging of Service Processes in Session 0Debugging Child ProcessesChild Debugging in ActionHow Child Debugging WorksChild Debugging as a “Startup Debugger” AlternativeExample: The Case of MSI Custom ActionsMore Useful TricksDebugging Error-Code FailuresDebugging Win32 API FailuresObserving Low-Level Resource Access Failures with the Process Monitor ToolDebugging High-Level, Error-Code FailuresTracking Down Error Codes the Brute-Force WayBreaking on First-Chance Exception NotificationsFreezing ThreadsFreezing and Unfreezing ThreadsThe Difference Between Freezing and Suspending ThreadsKernel-Mode Debugging TricksBreaking on User-Mode Process CreationThe nt!PspInsertProcess Kernel-Mode BreakpointBreaking on Instances of a Particular Process OnlyTracing Processes Started During the Windows Boot SequenceDebugging the Startup of User-Mode ProcessesBreaking on DLL Load EventsBreaking on Unhandled SEH ExceptionsFreezing ThreadsSummary
8. Common Debugging Scenarios, Part 1
Debugging Access ViolationsUnderstanding Memory Access ViolationsThe !analyze Debugger Extension CommandDebugging Heap CorruptionsDebugging Native Heap CorruptionsIntroducing the Page HeapHow Page Heap WorksEnabling Page Heap for a Target ProcessThe !heap Debugger Extension CommandCustomizing Page Heap for Your SituationDebugging Managed (GC) Heap CorruptionsThe !VerifyHeap SOS CommandManaged Debugging AssistantsDebugging Stack CorruptionsStack-Based Buffer OverrunsUsing Data Breakpoints in Stack Corruption InvestigationsReconstructing Call Frames from Corrupted StacksDebugging Stack OverflowsUnderstanding Stack OverflowsThe kf Debugger CommandDebugging Handle LeaksA Handle Leak ExampleThe !htrace Debugger Extension CommandHow !htrace Works!htrace in ActionLimitations of Handle-Leak Detection Using !htraceDebugging User-Mode Memory LeaksDetecting Resource Leaks Using the Application Verifier ToolInvestigating Memory Leaks Using the UMDH ToolHow UMDH WorksUMDH in ActionResolving Symbols in UMDHExtending the Strategy: A Custom Reference Stack-Trace DatabaseDebugging Kernel-Mode Memory LeaksKernel Memory BasicsInvestigating Kernel-Mode Leaks Using Pool TaggingSummary
9. Common Debugging Scenarios, Part 2
Debugging Race ConditionsShared-State Consistency BugsStress Testing and Reproducing Race ConditionsSimulating Race Conditions Under the DebuggerShared-State Consistency Bugs: A Few Suggested FixesShared-State Lifetime Management BugsThread Context Lifetime and Reference CountingGlobal Variables in Worker Threads and the Process Rundown SequenceDLL Module Lifetime-Management BugsDebugging Unloaded CodeDLL Module Lifetime-Management Bugs: A Few Suggested FixesDebugging DeadlocksLock-Ordering DeadlocksDebugging the Deadlock ConditionThe !cs Debugger Extension CommandAvoiding Lock-Ordering DeadlocksLogical DeadlocksDebugging the Deadlock ConditionThe Loader Lock and DllMain HazardsThe Loader Lock and the Debugger Break-in SequenceDebugging Access-Check ProblemsThe Basic NT Security ModelAnatomy of an Access CheckThe !token Debugger Extension CommandThe !sd Debugger Extension CommandWindows Vista ImprovementsIntegrity LevelsUAC in ActionWrapping UpSummary
10. Debugging System Internals
The Windows Console SubsystemThe Magic Behind printfThe Console Host Process in Windows 7The C Runtime Library’s I/O FunctionsHandling of Windows UI EventsHandling of the Ctrl+C SignalConsole Termination Handlers in .NET/Win32Console Termination Handlers at the System LevelAnatomy of System CallsThe User-Mode Side of System CallsThe Transition into Kernel ModeThe Kernel-Mode Side of System CallsSummary
3. Observing and Analyzing Software Behavior
11. Introducing Xperf
Acquiring XperfInstalling the Windows Performance ToolkitInstalling the Windows Performance ToolkitInstalling the Windows Performance ToolkitYour First Xperf InvestigationDevising an Investigation StrategyCollecting an ETW Trace for the ScenarioAnalyzing the Collected ETW TracePhase 1: Narrowing the List of BottlenecksPhase 2: Getting Stack Traces for the Hot SpotsXperf’s Strengths and LimitationsSummary
12. Inside ETW
ETW ArchitectureETW Design PrinciplesETW ComponentsThe Special NT Kernel Logger SessionConfiguring ETW Sessions Using XperfFilled Buffers and Lost ETW EventsUsing Circular ETW LoggingExisting ETW Instrumentation in WindowsInstrumentation in the Windows KernelThe PROC_THREAD and LOADER Kernel FlagsOther Kernel Flags and GroupsViewing Kernel Provider Events in XperfInstrumentation in Other Windows ComponentsManifest-Based User ProvidersDiscovering User ProvidersViewing User Provider Events in XperfUnderstanding ETW Stack-Walk EventsEnabling and Viewing Stack Traces for Kernel Provider EventsEnabling and Viewing Stack Traces for User Provider EventsDiagnosing ETW Stack-Trace IssuesDiagnosing Missing Call StacksDiagnosing Unresolved Call StacksDiagnosing Incomplete Call StacksAdding ETW Logging to Your CodeAnatomy of ETW EventsLogging Events Using the ETW Win32 APIsBoot Tracing in ETWLogging Kernel Provider Events During BootLogging User Provider Events During BootSummary
13. Common Tracing Scenarios
Analyzing Blocked TimeThe CSwitch and ReadyThread ETW EventsWait Analysis Using Visual Studio 2010Starting a Parallel Performance Analyzer Tracing SessionStarting a Parallel Performance Analyzer Tracing SessionWait Analysis Using XperfAnalyzing Memory UsageAnalyzing High-Level Memory Usage in a Target ProcessAnalyzing NT Heap Memory UsageCapturing ETW Heap Traces Using XperfAnalyzing ETW Heap Traces Using XperfAnalyzing GC Heap (.NET) Memory UsageAnalyzing GC Heap Memory Using PerfViewCollecting ETW Traces Using PerfViewAnalyzing GC Heap Memory Using SOSTracing as a Debugging AidTracing Error Code FailuresWhy Trace When You Can Debug?Tracing System Call FailuresTracing System InternalsDevising an Exploration StrategyObserving the Scenario Using Kernel ETW EventsObserving the Scenario Using User ETW EventsSummary
A. WinDbg User-Mode Debugging Quick Start
Starting a User-Mode Debugging SessionFixing the Symbols PathFixing the Sources PathDisplaying the Command Line of the Target ProcessControl Flow CommandsListing Loaded Modules and Their VersionResolving Function AddressesSetting Code (Software) BreakpointsSetting Data (Hardware) BreakpointsSwitching Between ThreadsDisplaying Call StacksDisplaying Function ParametersDisplaying Local VariablesDisplaying Data Members of Native TypesNavigating Between Call FramesListing Function DisassemblyDisplaying and Modifying Memory and Register ValuesEnding a User-Mode Debugging Session
B. WinDbg Kernel-Mode Debugging Quick Start
Starting a Kernel-Mode Debugging SessionSwitching Between CPU ContextsDisplaying Process InformationDisplaying Thread InformationSwitching Process and Thread ContextsListing Loaded Modules and Their VersionSetting Code (Software) Breakpoints Inside Kernel-Mode CodeSetting Code (Software) Breakpoints Inside User-Mode CodeSetting Data (Hardware) BreakpointsEnding a Kernel-Mode Debugging Session
Index
About the Author
Copyright

Content preview from Inside Windows® Debugging

Inside Windows® Debugging

Tarik Soulami

Published by Microsoft Press

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9780735671348Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Inside Windows® Debugging

by Tarik Soulami