The Shellcoder's Handbook: Discovering and Exploiting Security Holes

The Shellcoder's Handbook: Discovering and Exploiting Security Holes

by Jack Koziol, Dave Aitel, David Litchfield, Chris Anley, Sinan noir Eren, Neel Mehta, Riley Hassell
Released April 2004
Publisher(s): Wiley
ISBN: 9780764544682

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

  • Examines where security holes come from, how to discover them, how hackers exploit them and take control of systems on a daily basis, and most importantly, how to close these security holes so they never occur again

  • A unique author team-a blend of industry and underground experts- explain the techniques that readers can use to uncover security holes in any software or operating system

  • Shows how to pinpoint vulnerabilities in popular operating systems (including Windows, Linux, and Solaris) and applications (including MS SQL Server and Oracle databases)

  • Details how to deal with discovered vulnerabilities, sharing some previously unpublished advanced exploits and techniques

Table of contents

  1. Copyright
  2. About the Authors
  3. Credits
  4. Acknowledgments
  5. I. Introduction to Exploitation: Linux on x86
    1. 1. Before You Begin
      1. 1.1. Basic Concepts
        1. 1.1.1. Memory Management
        2. 1.1.2. Assembly
        3. 1.1.3. Registers
      2. 1.2. Recognizing C++ Code Constructs in Assembly
      3. 1.3. Conclusion
    2. 2. Stack Overflows
      1. 2.1. Buffers
      2. 2.2. The Stack
        1. 2.2.1. Functions and the Stack
      3. 2.3. Overflowing Buffers on the Stack
        1. 2.3.1. Controlling EIP
      4. 2.4. Using an Exploit to Get Root Privileges
        1. 2.4.1. The Address Problem
        2. 2.4.2. The NOP Method
      5. 2.5. Defeating a Non-Executable Stack
        1. 2.5.1. Return to libc
      6. 2.6. Conclusion
    3. 3. Shellcode
      1. 3.1. Understanding System Calls
      2. 3.2. Writing Shellcode for the exit() Syscall
      3. 3.3. Injectable Shellcode
      4. 3.4. Spawning a Shell
      5. 3.5. Conclusion
    4. 4. Introduction to Format String Bugs
      1. 4.1. Prerequisites
      2. 4.2. What Is a Format String?
      3. 4.3. What Is a Format String Bug?
      4. 4.4. Format String Exploits
        1. 4.4.1. Crashing Services
        2. 4.4.2. Information Leakage
      5. 4.5. Controlling Execution for Exploitation
      6. 4.6. Why Did This Happen?
      7. 4.7. Format String Technique Roundup
      8. 4.8. Conclusion
    5. 5. Introduction to Heap Overflows
      1. 5.1. What Is a Heap?
        1. 5.1.1. How a Heap Works
      2. 5.2. Finding Heap Overflows
        1. 5.2.1. Basic Heap Overflows
        2. 5.2.2. Intermediate Heap Overflows
        3. 5.2.3. Advanced Heap Overflow Exploitation
          1. 5.2.3.1. What to Overwrite
            1. 5.2.3.1.1. GOT Entries
            2. 5.2.3.1.2. Global Function Pointers
            3. 5.2.3.1.3. DTORS
            4. 5.2.3.1.4. atexit Handlers
            5. 5.2.3.1.5. Stack Values
      3. 5.3. Conclusion
  6. II. Exploiting More Platforms: Windows, Solaris, and Tru64
    1. 6. The Wild World of Windows
      1. 6.1. How Does Windows Differ from Linux?
        1. 6.1.1. Win32 API and PE-COFF
      2. 6.2. Heaps
        1. 6.2.1. Threading
      3. 6.3. The Genius and Idiocy of the Distributed Common Object Model and DCE-RPC
        1. 6.3.1. Recon
        2. 6.3.2. Exploitation
        3. 6.3.3. Tokens and Impersonation
        4. 6.3.4. Exception Handling under Win32
      4. 6.4. Debugging Windows
        1. 6.4.1. Bugs in Win32
        2. 6.4.2. Writing Windows Shellcode
        3. 6.4.3. A Hacker's Guide to the Win32 API
        4. 6.4.4. A Windows Family Tree from the Hacker's Perspective
      5. 6.5. Conclusion
    2. 7. Windows Shellcode
      1. 7.1. Syntax and Filters
      2. 7.2. Setting Up
      3. 7.3. Parsing the PEB
        1. 7.3.1. Heapoverflow.c Analysis
      4. 7.4. Searching with Windows Exception Handling
      5. 7.5. Popping a Shell
        1. 7.5.1. Why You Should Never Pop a Shell on Windows
      6. 7.6. Conclusion
    3. 8. Windows Overflows
      1. 8.1. Stack-Based Buffer Overflows
        1. 8.1.1. Frame-Based Exception Handlers
        2. 8.1.2. Abusing Frame-Based Exception Handling on Windows 2003 Server
          1. 8.1.2.1. Abusing an Existing Handler
          2. 8.1.2.2. Find a block of code in an address not associated with a module that will get us back to our buffer
          3. 8.1.2.3. Find a block of code in the address space of a module that does not have a Load Configuration Directory
        3. 8.1.3. A Final Note about Frame-Based Handler Overwrites
      2. 8.2. Stack Protection and Windows 2003 Server
      3. 8.3. Heap-Based Buffer Overflows
        1. 8.3.1. The Process Heap
        2. 8.3.2. Dynamic Heaps
        3. 8.3.3. Working with the Heap
        4. 8.3.4. How the Heap Works
      4. 8.4. Exploiting Heap-Based Overflows
        1. 8.4.1. Overwrite Pointer to RtlEnterCriticalSection in the PEB
        2. 8.4.2. Overwrite Pointer to First Vectored Handler at 77FC3210
        3. 8.4.3. Overwrite Pointer to Unhandled Exception Filter
        4. 8.4.4. Overwrite Pointer to Exception Handler in Thread Environment Block
        5. 8.4.5. Repairing the Heap
        6. 8.4.6. Other Aspects of Heap-Based Overflows
          1. 8.4.6.1. COM Objects and the Heap
          2. 8.4.6.2. Overflowing Logic Program Control Data
        7. 8.4.7. Wrapping Up the Heap
      5. 8.5. Other Overflows
        1. 8.5.1. .data section overflows
        2. 8.5.2. TEB/PEB Overflows
      6. 8.6. Exploiting Buffer Overflows and Non-Executable Stacks
      7. 8.7. Conclusion
    4. 9. Overcoming Filters
      1. 9.1. Writing Exploits for Use with an Alphanumeric Filter
      2. 9.2. Writing Exploits for Use with a Unicode Filter
        1. 9.2.1. What Is Unicode?
        2. 9.2.2. Converting from ASCII to Unicode
      3. 9.3. Exploiting Unicode-Based Vulnerabilities
        1. 9.3.1. The Available Instruction Set in Unicode Exploits
      4. 9.4. The Venetian Method
        1. 9.4.1. An ASCII Venetian Implementation
      5. 9.5. Decoder and Decoding
        1. 9.5.1. The Decoder Code
        2. 9.5.2. Getting a Fix on the Buffer Address
      6. 9.6. Conclusion
    5. 10. Introduction to Solaris Exploitation
      1. 10.1. Introduction to the SPARC Architecture
        1. 10.1.1. Registers and Register Windows
        2. 10.1.2. The Delay Slot
        3. 10.1.3. Synthetic Instructions
      2. 10.2. Solaris/SPARC Shellcode Basics
        1. 10.2.1. Self-Location Determination and SPARC Shellcode
        2. 10.2.2. Simple SPARC exec Shellcode
        3. 10.2.3. Useful System Calls on Solaris
        4. 10.2.4. NOP and Padding Instructions
      3. 10.3. Solaris/SPARC Stack Frame Introduction
      4. 10.4. Stack-Based Overflow Methodologies
        1. 10.4.1. Arbitrary Size Overflow
        2. 10.4.2. Register Windows and Stack Overflow Complications
        3. 10.4.3. Other Complicating Factors
        4. 10.4.4. Possible Solutions
        5. 10.4.5. Off-By-One Stack Overflow Vulnerabilities
        6. 10.4.6. Shellcode Locations
      5. 10.5. Stack Overflow Exploitation In Action
        1. 10.5.1. The Vulnerable Program
        2. 10.5.2. The Exploit
      6. 10.6. Heap-Based Overflows on Solaris/SPARC
        1. 10.6.1. Solaris System V Heap Introduction
        2. 10.6.2. Heap Tree Structure
      7. 10.7. Basic Exploit Methodology (t_delete)
        1. 10.7.1. Standard Heap Overflow Limitations
        2. 10.7.2. Targets for Overwrite
          1. 10.7.2.1. The Bottom Chunk
          2. 10.7.2.2. Small Chunk Corruption
      8. 10.8. Other Heap-Related Vulnerabilities
        1. 10.8.1. Off-by-One Overflows
        2. 10.8.2. Double Free Vulnerabilities
        3. 10.8.3. Arbitrary Free Vulnerabilities
      9. 10.9. Heap Overflow Example
        1. 10.9.1. The Vulnerable Program
      10. 10.10. Other Solaris Exploitation Techniques
        1. 10.10.1. Static Data Overflows
        2. 10.10.2. Bypassing the Non-Executable Stack Protection
      11. 10.11. Conclusion
    6. 11. Advanced Solaris Exploitation
      1. 11.1. Single Stepping the Dynamic Linker
      2. 11.2. Various Style Tricks for Solaris SPARC Heap Overflows
      3. 11.3. Advanced Solaris/SPARC Shellcode
      4. 11.4. Conclusion
    7. 12. HP Tru64 Unix Exploitation
      1. 12.1. The Alpha Architecture
        1. 12.1.1. Alpha Registers
        2. 12.1.2. Instruction Set
        3. 12.1.3. Calling Conventions
      2. 12.2. Retrieving the Program Counter (GetPC)
      3. 12.3. System Call Invocation
      4. 12.4. XOR Decoder for Shellcode
      5. 12.5. .end main setuid + execve Shellcode
        1. 12.5.1. Code the setuid(0) + execve("/bin/sh", . . .) systemcalls
        2. 12.5.2. Compile the Assembly Code and Extract the Main Function
        3. 12.5.3. Encode the Extracted opcodes with the XOR Key
        4. 12.5.4. Plug the Encoded Code into the XOR Decoder
        5. 12.5.5. Compile and Extract the Final Shellcode
      6. 12.6. Connect-Back Shellcode
      7. 12.7. Find-Socket Shellcode
      8. 12.8. Bind-Socket Shellcode
      9. 12.9. Stack Overflow Exploitation
        1. 12.9.1. Defeating the Non-Executable Stack
      10. 12.10. Exploiting rpc.ttdbserver
      11. 12.11. Conclusion
  7. III. Vulnerability Discovery
    1. 13. Establishing a Working Environment
      1. 13.1. What You Need for Reference
      2. 13.2. What You Need for Code
        1. 13.2.1. gcc
        2. 13.2.2. gdb
        3. 13.2.3. NASM
        4. 13.2.4. WinDbg
        5. 13.2.5. OllyDbg
        6. 13.2.6. SoftICE
        7. 13.2.7. Visual C++
        8. 13.2.8. Python
      3. 13.3. What You Need for Investigation
        1. 13.3.1. Useful Custom Scripts/Tools
          1. 13.3.1.1. An Offset Finder
          2. 13.3.1.2. Generic Fuzzers
          3. 13.3.1.3. The Debug Trick
        2. 13.3.2. All Platforms
        3. 13.3.3. Unix
          1. 13.3.3.1. ltrace
          2. 13.3.3.2. strace
          3. 13.3.3.3. fstat (BSD)
          4. 13.3.3.4. tcpdump
          5. 13.3.3.5. Ethereal
        4. 13.3.4. Windows
          1. 13.3.4.1. IDA Pro Disassembler
      4. 13.4. What You Need to Know
        1. 13.4.1. Paper Archives
      5. 13.5. Optimizing Shellcode Development
        1. 13.5.1. Plan the Exploit
        2. 13.5.2. Write the Shellcode in Inline Assembler
        3. 13.5.3. Maintain a Shellcode Library
        4. 13.5.4. Make It Continue Nicely
        5. 13.5.5. Make the Exploit Stable
        6. 13.5.6. Make It Steal the Connection
      6. 13.6. Conclusion
    2. 14. Fault Injection
      1. 14.1. Design Overview
        1. 14.1.1. Input Generation
          1. 14.1.1.1. Manual Generation
          2. 14.1.1.2. Automated Generation
          3. 14.1.1.3. Live Capture
          4. 14.1.1.4. "Fuzz" Generation
        2. 14.1.2. Fault Injection
        3. 14.1.3. Modification Engines
          1. 14.1.3.1. Delimiting Logic
          2. 14.1.3.2. Getting around Input Sanitization
        4. 14.1.4. Fault Delivery
        5. 14.1.5. Nagel Algorithm
        6. 14.1.6. Timing
        7. 14.1.7. Heuristics
        8. 14.1.8. Stateless versus State-Based Protocols
      2. 14.2. Fault Monitoring
        1. 14.2.1. Using a Debugger
        2. 14.2.2. FaultMon
      3. 14.3. Putting It Together
      4. 14.4. Conclusion
    3. 15. The Art of Fuzzing
      1. 15.1. General Theory of Fuzzing
        1. 15.1.1. Static Analysis versus Fuzzing
        2. 15.1.2. Fuzzing Is Scalable
      2. 15.2. Weaknesses in Fuzzers
      3. 15.3. Modeling Arbitrary Network Protocols
      4. 15.4. Other Fuzzer Possibilities
        1. 15.4.1. Bit Flipping
        2. 15.4.2. Modifying Open Source Programs
        3. 15.4.3. Fuzzing with Dynamic Analysis
      5. 15.5. SPIKE
        1. 15.5.1. What Is a Spike?
        2. 15.5.2. Why Use the SPIKE Data Structure to Model Network Protocols?
          1. 15.5.2.1. Various Programs Included with SPIKE
          2. 15.5.2.2. SPIKE Example: dtlogin
      6. 15.6. Other Fuzzers
      7. 15.7. Conclusion
    4. 16. Source Code Auditing: Finding Vulnerabilities in C-Based Languages
      1. 16.1. Tools
        1. 16.1.1. Cscope
        2. 16.1.2. Ctags
        3. 16.1.3. Editors
        4. 16.1.4. Cbrowser
      2. 16.2. Automated Source Code Analysis Tools
      3. 16.3. Methodology
        1. 16.3.1. Top-Down (Specific) Approach
        2. 16.3.2. Bottom-Up Approach
        3. 16.3.3. Selective Approach
      4. 16.4. Vulnerability Classes
        1. 16.4.1. Generic Logic Errors
        2. 16.4.2. (Almost) Extinct Bug Classes
        3. 16.4.3. Format Strings
        4. 16.4.4. Generic Incorrect Bounds-Checking
        5. 16.4.5. Loop Constructs
        6. 16.4.6. Off-by-One Vulnerabilities
        7. 16.4.7. Non-Null Termination Issues
        8. 16.4.8. Skipping Null-Termination Issues
        9. 16.4.9. Signed Comparison Vulnerabilities
        10. 16.4.10. Integer-Related Vulnerabilities
        11. 16.4.11. Different-Sized Integer Conversions
        12. 16.4.12. Double Free Vulnerabilities
        13. 16.4.13. Out-of-Scope Memory Usage Vulnerabilities
        14. 16.4.14. Uninitialized Variable Usage
        15. 16.4.15. Use After Free Vulnerabilities
        16. 16.4.16. Multithreaded Issues and Re-Entrant Safe Code
      5. 16.5. Beyond Recognition: A Real Vulnerability versus a Bug
      6. 16.6. Conclusion
    5. 17. Instrumented Investigation: A Manual Approach
      1. 17.1. Philosophy
      2. 17.2. Oracle extproc Overflow
      3. 17.3. Common Architectural Failures
        1. 17.3.1. Problems Happen at Boundaries
          1. 17.3.1.1. A Process Calling into an External Process on the Same Host
          2. 17.3.1.2. A Process Calling into an External, Dynamically Loaded Library
          3. 17.3.1.3. A Process Calling into a Function on a Remote Host
        2. 17.3.2. Problems Happen When Data Is Translated
        3. 17.3.3. Problems Cluster in Areas of Asymmetry
        4. 17.3.4. Problems Occur When Authentication and Authorization Are Confused
        5. 17.3.5. Problems Occur in the Dumbest Places
      4. 17.4. Bypassing Input Validation and Attack Detection
        1. 17.4.1. Stripping Bad Data
        2. 17.4.2. Using Alternate Encodings
        3. 17.4.3. Using File-Handling Features
          1. 17.4.3.1. Required String Is Present in Path
          2. 17.4.3.2. Prohibited String Not Present in Path
          3. 17.4.3.3. Incorrect Behavior Based on File Extension
        4. 17.4.4. Evading Attack Signatures
        5. 17.4.5. Defeating Length Limitations
          1. 17.4.5.1. Sea Monkey Data
          2. 17.4.5.2. Harmful Truncation—Severing Escape Characters
          3. 17.4.5.3. Multiple Attempts
          4. 17.4.5.4. Context-Free Length Limits
      5. 17.5. Windows 2000 SNMP DOS
      6. 17.6. Finding DOS Attacks
      7. 17.7. SQL-UDP
      8. 17.8. Conclusion
    6. 18. Tracing for Vulnerabilities
      1. 18.1. Overview
        1. 18.1.1. A Vulnerable Program
        2. 18.1.2. Component Design
          1. 18.1.2.1. Process Injection
          2. 18.1.2.2. Machine-Code Analysis
            1. 18.1.2.2.1. Static Linking
            2. 18.1.2.2.2. Importing
            3. 18.1.2.2.3. Inlining
          3. 18.1.2.3. Function Hooking
            1. 18.1.2.3.1. Import Hooking
            2. 18.1.2.3.2. Prelude Hooking
            3. 18.1.2.3.3. Prologue Hooking
          4. 18.1.2.4. Data Collection
        3. 18.1.3. Building VulnTrace
          1. 18.1.3.1. VTInject
          2. 18.1.3.2. VulnTrace.dll
        4. 18.1.4. Using VulnTrace
        5. 18.1.5. Advanced Techniques
          1. 18.1.5.1. Fingerprint Systems
          2. 18.1.5.2. More Vulnerability Classes
            1. 18.1.5.2.1. Integer Overflows
            2. 18.1.5.2.2. Format Bugs
            3. 18.1.5.2.3. Other Classes
      2. 18.2. Conclusion
    7. 19. Binary Auditing: Hacking Closed Source Software
      1. 19.1. Binary versus Source-Code Auditing: The Obvious Differences
      2. 19.2. IDA Pro—The Tool of the Trade
        1. 19.2.1. Features: A Quick Crash Course
        2. 19.2.2. Debugging Symbols
      3. 19.3. Binary Auditing Introduction
        1. 19.3.1. Stack Frames
          1. 19.3.1.1. Traditional BP-Based Stack Frames
          2. 19.3.1.2. Functions without a Frame Pointer
          3. 19.3.1.3. Non-Traditional BP-Based Stack Frames
        2. 19.3.2. Calling Conventions
          1. 19.3.2.1. The C Calling Convention
          2. 19.3.2.2. The Stdcall Calling Convention
        3. 19.3.3. Compiler-Generated Code
          1. 19.3.3.1. Function Layouts
          2. 19.3.3.2. If Statements
          3. 19.3.3.3. For and While Loops
          4. 19.3.3.4. Switch Statements
        4. 19.3.4. memcpy-Like Code Constructs
        5. 19.3.5. strlen-Like Code Constructs
        6. 19.3.6. C++ Code Constructs
        7. 19.3.7. The this Pointer
      4. 19.4. Reconstructing Class Definitions
        1. 19.4.1. vtables
        2. 19.4.2. Quick but Useful Tidbits
      5. 19.5. Manual Binary Analysis
        1. 19.5.1. Quick Examination of Library Calls
        2. 19.5.2. Suspicious Loops and Write Instructions
        3. 19.5.3. Higher-Level Understanding and Logic Bugs
        4. 19.5.4. Graphical Analysis of Binaries
        5. 19.5.5. Manual Decompilation
      6. 19.6. Binary Vulnerability Examples
        1. 19.6.1. Microsoft SQL Server Bugs
        2. 19.6.2. LSD's RPC-DCOM Vulnerability
        3. 19.6.3. IIS WebDAV Vulnerability
      7. 19.7. Conclusion
  8. IV. Advanced Materials
    1. 20. Alternative Payload Strategies
      1. 20.1. Modifying the Program
      2. 20.2. The SQL Server 3-Byte Patch
      3. 20.3. The MySQL 1-Bit Patch
      4. 20.4. OpenSSH RSA Authentication Patch
      5. 20.5. Other Runtime Patching Ideas
        1. 20.5.1. GPG 1.2.2 Randomness Patch
      6. 20.6. Upload and Run (or Proglet Server)
      7. 20.7. Syscall Proxies
      8. 20.8. Problems with Syscall Proxies
      9. 20.9. Conclusion
    2. 21. Writing Exploits that Work in the Wild
      1. 21.1. Factors in Unreliability
        1. 21.1.1. Magic Numbers
        2. 21.1.2. Versioning
        3. 21.1.3. Shellcode Problems
          1. 21.1.3.1. Network Related
          2. 21.1.3.2. Privilege Related
          3. 21.1.3.3. Configuration Related
          4. 21.1.3.4. Host IDS Related
          5. 21.1.3.5. Thread Related
      2. 21.2. Countermeasures
        1. 21.2.1. Preparation
        2. 21.2.2. Brute Forcing
        3. 21.2.3. Local Exploits
        4. 21.2.4. OS/Application Fingerprinting
        5. 21.2.5. Information Leaks
      3. 21.3. Conclusion
    3. 22. Attacking Database Software
      1. 22.1. Network Layer Attacks
      2. 22.2. Application Layer Attacks
      3. 22.3. Running Operating System Commands
        1. 22.3.1. Microsoft SQL Server
        2. 22.3.2. Oracle
        3. 22.3.3. IBM DB2
      4. 22.4. Exploiting Overruns at the SQL Level
        1. 22.4.1. SQL Functions
          1. 22.4.1.1. Using the CHR/CHAR Function
      5. 22.5. Conclusion
    4. 23. Kernel Overflows
      1. 23.1. Kernel Vulnerability Types
      2. 23.2. 0day Kernel Vulnerabilities
        1. 23.2.1. OpenBSD exec_ibcs2_coff_prep_zmagic() Stack Overflow
        2. 23.2.2. The Vulnerability
      3. 23.3. Solaris vfs_getvfssw() Loadable Kernel Module Traversal Vulnerability
        1. 23.3.1. The sysfs() System Call
        2. 23.3.2. The mount() System Call
      4. 23.4. Conclusion
    5. 24. Exploiting Kernel Vulnerabilities
      1. 24.1. The exec_ibcs2_coff_prep_zmagic() Vulnerability
        1. 24.1.1. Calculating Offsets and Breakpoints
        2. 24.1.2. Overwriting the Return Address and Redirecting Execution
        3. 24.1.3. Locating the Process Descriptor (or the Proc Structure)
          1. 24.1.3.1. Stack Lookup
          2. 24.1.3.2. sysctl() Syscall
        4. 24.1.4. Kernel Mode Payload Creation
          1. 24.1.4.1. p_cred and u_cred
          2. 24.1.4.2. Breaking chroot
        5. 24.1.5. Returning Back from Kernel Payload
          1. 24.1.5.1. Return to User Mode: iret Technique
          2. 24.1.5.2. Return to Kernel Code: sidt Technique and _kernel_text Search
        6. 24.1.6. Getting root (uid=0)
      2. 24.2. Solaris vfs_getvfssw() Loadable Kernel Module Path Traversal Exploit
        1. 24.2.1. Crafting the Exploit
        2. 24.2.2. The Kernel Module to Load
        3. 24.2.3. Getting root (uid=0)
      3. 24.3. Conclusion

Product information

  • Title: The Shellcoder's Handbook: Discovering and Exploiting Security Holes
  • Author(s): Jack Koziol, Dave Aitel, David Litchfield, Chris Anley, Sinan noir Eren, Neel Mehta, Riley Hassell
  • Release date: April 2004
  • Publisher(s): Wiley
  • ISBN: 9780764544682