book

The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition

Name: The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition
ISBN: 9780470080238

by Chris Anley, John Heasman, Felix FX Lindner, Gerardo Richarte

August 2007

Intermediate to advanced

743 pages

16h 50m

English

Wiley

Read now

Unlock full access

Copyright
About the Authors
Credits
Acknowledgments
Introduction to the Second Edition
I. Introduction to Exploitation: Linux on x86
1. Before You Begin
1.1. Basic Concepts1.1.1. Memory Management1.1.2. Assembly1.1.2.1. Registers1.2. Recognizing C and C++ Code Constructs in Assembly1.3. Conclusion
2. Stack Overflows
2.1. Buffers2.2. The Stack2.2.1. Functions and the Stack2.3. Overflowing Buffers on the Stack2.3.1. Controlling EIP2.4. An Interesting Diversion2.5. Using an Exploit to Get Root Privileges2.5.1. The Address Problem2.5.2. The NOP Method2.6. Defeating a Non-Executable Stack2.6.1. Return to libc2.7. Conclusion
3. Shellcode
3.1. Understanding System Calls3.2. Writing Shellcode for the exit() Syscall3.3. Injectable Shellcode3.4. Spawning a Shell3.5. Conclusion
4. Introduction to Format String Bugs
4.1. Prerequisites4.2. What Is a Format String?4.3. What Is a Format String Bug?4.4. Format String Exploits4.4.1. Crashing Services4.4.2. Information Leakage4.5. Controlling Execution for Exploitation4.6. Why Did This Happen?4.7. Format String Technique Roundup4.8. Conclusion

5. Introduction to Heap Overflows
5.1. What Is a Heap?5.1.1. How a Heap Works5.2. Finding Heap Overflows5.2.1. Basic Heap Overflows5.2.2. Intermediate Heap Overflows5.2.3. Advanced Heap Overflow Exploitation5.2.3.1. What to Overwrite5.2.3.1.1. GOT Entries5.2.3.1.2. Global Function Pointers5.2.3.1.3. DTORS5.2.3.1.4. atexit Handlers5.2.3.1.5. Stack Values5.3. Conclusion
II. Other Platforms—Windows, Solaris, OS/X, and Cisco
6. The Wild World of Windows
6.1. How Does Windows Differ from Linux?6.1.1. Win32 API and PE-COFF6.2. Heaps6.2.1. Threading6.3. The Genius and Idiocy of the Distributed Common Object Model and DCE-RPC6.3.1. Recon6.3.2. Exploitation6.3.3. Tokens and Impersonation6.3.4. Exception Handling under Win326.4. Debugging Windows6.4.1. Bugs in Win326.4.2. Writing Windows Shellcode6.4.3. A Hacker's Guide to the Win32 API6.4.4. A Windows Family Tree from the Hacker's Perspective6.5. Conclusion
7. Windows Shellcode
7.1. Syntax and Filters7.2. Setting Up7.2.1. Parsing the PEB7.2.2. Heapoverflow.c Analysis7.2.3. Searching with Windows Exception Handling7.3. Popping a Shell7.4. Why You Should Never Pop a Shell on Windows7.5. Conclusion
8. Windows Overflows
8.1. Stack-Based Buffer Overflows8.2. Frame-Based Exception Handlers8.3. Abusing Frame-Based Exception Handling on Windows 2003 Server8.3.1.8.3.1.1. Abusing an Existing Handler8.3.1.2. Find a Block of Code in an Address Not Associated with a Module That Will Get Us Back to Our Buffer8.3.1.3. Find a Block of Code in the Address Space of a Module That Does Not Have a Load Configuration Directory8.3.2. A Final Note about Frame-Based Handler Overwrites8.4. Stack Protection and Windows 2003 Server8.5. Heap-Based Buffer Overflows8.6. The Process Heap8.6.1. Dynamic Heaps8.6.2. Working with the Heap8.6.3. How the Heap Works8.7. Exploiting Heap-Based Overflows8.7.1. Overwrite Pointer to RtlEnterCriticalSection in the PEB8.7.2. Overwrite Pointer to Unhandled Exception Filter8.7.3. Repairing the Heap8.7.4. Other Aspects of Heap-Based Overflows8.7.4.1. COM Objects and the Heap8.7.4.2. Overflowing Logic Program Control Data8.7.5. Wrapping Up the Heap8.8. Other Overflows8.8.1. .data Section Overflows8.8.2. TEB/PEB Overflows8.9. Exploiting Buffer Overflows and Non-Executable Stacks8.10. Conclusion
9. Overcoming Filters
9.1. Writing Exploits for Use with an Alphanumeric Filter9.2. Writing Exploits for Use with a Unicode Filter9.2.1. What Is Unicode?9.2.2. Converting from ASCII to Unicode9.3. Exploiting Unicode-Based Vulnerabilities9.3.1. The Available Instruction Set in Unicode Exploits9.4. The Venetian Method9.4.1. An ASCII Venetian Implementation9.5. Decoder and Decoding9.5.1. The Decoder Code9.5.2. Getting a Fix on the Buffer Address9.6. Conclusion
10. Introduction to Solaris Exploitation
10.1. Introduction to the SPARC Architecture10.1.1. Registers and Register Windows10.1.2. The Delay Slot10.1.3. Synthetic Instructions10.2. Solaris/SPARC Shellcode Basics10.2.1. Self-Location Determination and SPARC Shellcode10.2.2. Simple SPARC exec Shellcode10.2.3. Useful System Calls on Solaris10.2.4. NOP and Padding Instructions10.3. Solaris/SPARC Stack Frame Introduction10.4. Stack-Based Overflow Methodologies10.4.1. Arbitrary Size Overflow10.4.2. Register Windows and Stack Overflow Complications10.4.3. Other Complicating Factors10.4.4. Possible Solutions10.4.5. Off-By-One Stack Overflow Vulnerabilities10.4.6. Shellcode Locations10.5. Stack Overflow Exploitation In Action10.5.1. The Vulnerable Program10.5.2. The Exploit10.6. Heap-Based Overflows on Solaris/SPARC10.6.1. Solaris System V Heap Introduction10.6.2. Heap Tree Structure10.7. Basic Exploit Methodology (t_delete)10.7.1. Standard Heap Overflow Limitations10.7.2. Targets for Overwrite10.7.2.1. The Bottom Chunk10.7.2.2. Small Chunk Corruption10.8. Other Heap-Related Vulnerabilities10.8.1. Off-by-One Overflows10.8.2. Double Free Vulnerabilities10.8.3. Arbitrary Free Vulnerabilities10.9. Heap Overflow Example10.9.1. The Vulnerable Program10.10. Other Solaris Exploitation Techniques10.10.1. Static Data Overflows10.10.2. Bypassing the Non-Executable Stack Protection10.11. Conclusion
11. Advanced Solaris Exploitation
11.1. Single Stepping the Dynamic Linker11.2. Various Style Tricks for Solaris SPARC Heap Overflows11.3. Advanced Solaris/SPARC Shellcode11.4. Conclusion
12. OS X Shellcode
12.1. OS X Is Just BSD, Right?12.2. Is OS X Open Source?12.3. OS X for the Unix-aware12.3.1. Password Cracking12.4. OS X PowerPC Shellcode12.5. OS X Intel Shellcode12.5.1. Example Shellcode12.5.2. ret2libc12.5.3. ret2str(l)cpy12.6. OS X Cross-Platform Shellcode12.7. OS X Heap Exploitation12.8. Bug Hunting on OS X12.9. Some Interesting Bugs12.10. Essential Reading for OS X Exploits12.11. Conclusion
13. Cisco IOS Exploitation
13.1. An Overview of Cisco IOS13.1.1. Hardware Platforms13.1.2. Software Packages13.1.3. IOS System Architecture13.1.3.1. Memory Layout13.1.3.2. IOS Heap13.1.3.3. IO Memory13.2. Vulnerabilities in Cisco IOS13.2.1. Protocol Parsing Code13.2.2. Services on the Router13.2.3. Security Features13.2.4. The Command-Line Interface13.3. Reverse Engineering IOS13.3.1. Taking the Images Apart13.3.2. Diffing IOS Images13.3.3. Runtime Analysis13.3.3.1. Cisco Onboard Tools13.3.3.1.1. The ROM Monitor13.3.3.1.2. Crash Dumps13.3.3.2. GDB Agent13.4. Exploiting Cisco IOS13.4.1. Stack Overflows13.4.2. Heap Overflows13.4.2.1. IOS-Specific Challenges13.4.2.2. Memory Write on Unlink13.4.2.3. Alternatives13.4.2.4. Partial Attacks13.4.2.4.1. NVRAM Invalidation13.4.2.4.2. Global Variable Overwrite13.4.3. Shellcodes13.4.3.1. Configuration Changing Shellcode13.4.3.1.1. Complete Replacement13.4.3.1.2. Partial Replacement13.4.3.2. Runtime Image Patching Shellcode13.4.3.3. Bind Shell13.5. Conclusion
14. Protection Mechanisms
14.1. Protections14.1.1. Non-Executable Stack14.1.2. W^X (Either Writable or Executable) Memory14.1.3. Stack Data Protection14.1.3.1. Canaries14.1.3.2. Ideal Stack Layout14.1.4. AAAS: ASCII Armored Address Space14.1.5. ASLR: Address Space Layout Randomization14.1.6. Heap Protections14.1.7. Windows SEH Protections14.1.7.1. EEREAP14.1.7.2. pdest14.1.7.3. SEHInspector14.1.8. Other Protections14.1.8.1. Kernel Protections14.1.8.2. Pointer Protections14.2. Implementation Differences14.2.1. Windows14.2.1.1. W^X14.2.1.2. ASLR14.2.1.3. Stack Data Protections14.2.1.4. Heap Protections14.2.1.5. SEH Protections14.2.1.6. Others14.2.2. Linux14.2.2.1. W^X14.2.2.2. ASLR14.2.2.3. Stack Data Protections14.2.2.4. Heap Protections14.2.2.5. Others14.2.3. OpenBSD14.2.3.1. W^X14.2.3.2. ASLR14.2.3.3. Stack Data Protections14.2.3.4. Heap Protections14.2.3.5. Others14.2.4. Mac OS X14.2.4.1. W^X14.2.4.2. ASLR14.2.4.3. Stack Data Protections14.2.4.4. Heap Protections14.2.4.5. Others14.2.5. Solaris14.2.5.1. W^X14.2.5.2. ASLR14.2.5.3. Stack Data Protections14.2.5.4. Heap Protections14.2.5.5. Others14.3. Conclusion
III. Vulnerability Discovery
15. Establishing a Working Environment
15.1. What You Need for Reference15.2. What You Need for Code15.2.1. gcc15.2.2. gdb15.2.3. NASM15.2.4. WinDbg15.2.5. OllyDbg15.2.6. Visual C++15.2.7. Python15.3. What You Need for Investigation15.3.1. Useful Custom Scripts/Tools15.3.2.15.3.1.1. An Offset Finder15.3.1.2. Generic Fuzzers15.3.1.3. The Debug Trick15.3.2. All Platforms15.3.3. Unix15.3.3.1. ltrace and strace15.3.3.2. truss15.3.3.3. fstat (BSD)15.3.3.4. tcpdump15.3.3.5. Wireshark (formerly Ethereal)15.3.4. Windows15.3.4.1. IDA Pro Disassembler15.4. What You Need to Know15.4.1. Paper Archives15.5. Optimizing Shellcode Development15.5.1. Plan the Exploit15.5.2. Write the Shellcode in Inline Assembler15.5.3. Maintain a Shellcode Library15.5.4. Make It Continue Nicely15.5.5. Make the Exploit Stable15.5.6. Make It Steal the Connection15.6. Conclusion
16. Fault Injection
16.1. Design Overview16.1.1. Input Generation16.1.1.1. Manual Generation16.1.1.2. Automated Generation16.1.1.3. Live Capture16.1.1.4. "Fuzz" Generation16.1.2. Fault Injection16.1.3. Modification Engines16.1.3.1. Delimiting Logic16.1.3.2. Getting around Input Sanitization16.1.4. Fault Delivery16.1.5. Nagel Algorithm16.1.6. Timing16.1.7. Heuristics16.1.8. Stateless versus State-Based Protocols16.2. Fault Monitoring16.2.1. Using a Debugger16.2.2. FaultMon16.3. Putting It Together16.4. Conclusion
17. The Art of Fuzzing
17.1. General Theory of Fuzzing17.1.1. Static Analysis versus Fuzzing17.1.2. Fuzzing Is Scalable17.2. Weaknesses in Fuzzers17.3. Modeling Arbitrary Network Protocols17.4. Other Fuzzer Possibilities17.4.1. Bit Flipping17.4.2. Modifying Open Source Programs17.4.3. Fuzzing with Dynamic Analysis17.5. SPIKE17.5.1. What Is a Spike?17.5.2. Why Use the SPIKE Data Structure to Model Network Protocols?17.5.2.1. Various Programs Included with SPIKE17.5.2.2. SPIKE Example: dtlogin17.6. Other Fuzzers17.7. Conclusion
18. Source Code Auditing: Finding Vulnerabilities in C-Based Languages
18.1. Tools18.1.1. Cscope18.1.2. Ctags18.1.3. Editors18.1.4. Cbrowser18.2. Automated Source Code Analysis Tools18.3. Methodology18.3.1. Top-Down (Specific) Approach18.3.2. Bottom-Up Approach18.3.3. Selective Approach18.4. Vulnerability Classes18.4.1. Generic Logic Errors18.4.2. (Almost) Extinct Bug Classes18.4.3. Format Strings18.4.4. Generic Incorrect Bounds-Checking18.4.5. Loop Constructs18.4.6. Off-by-One Vulnerabilities18.4.7. Non-Null Termination Issues18.4.8. Skipping Null-Termination Issues18.4.9. Signed Comparison Vulnerabilities18.4.10. Integer-Related Vulnerabilities18.4.11. Different-Sized Integer Conversions18.4.12. Double Free Vulnerabilities18.4.13. Out-of-Scope Memory Usage Vulnerabilities18.4.14. Uninitialized Variable Usage18.4.15. Use After Free Vulnerabilities18.4.16. Multithreaded Issues and Re-Entrant Safe Code18.5. Beyond Recognition: A Real Vulnerability versus a Bug18.6. Conclusion
19. Instrumented Investigation: A Manual Approach
19.1. Philosophy19.2. Oracle extproc Overflow19.3. Common Architectural Failures19.3.1. Problems Happen at Boundaries19.3.1.1. A Process Calling into an External Process on the Same Host19.3.1.2. A Process Calling into an External, Dynamically Loaded Library19.3.1.3. A Process Calling into a Function on a Remote Host19.3.2. Problems Happen When Data Is Translated19.3.3. Problems Cluster in Areas of Asymmetry19.3.4. Problems Occur When Authentication and Authorization Are Confused19.3.5. Problems Occur in the Dumbest Places19.4. Bypassing Input Validation and Attack Detection19.4.1. Stripping Bad Data19.4.2. Using Alternate Encodings19.4.3. Using File-Handling Features19.4.3.1. Required String Is Present in Path19.4.3.2. Prohibited String Not Present in Path19.4.3.3. Incorrect Behavior Based on File Extension19.4.4. Evading Attack Signatures19.4.5. Defeating Length Limitations19.4.5.1. Sea Monkey Data19.4.5.2. Harmful Truncation —Severing Escape Characters19.4.5.3. Multiple Attempts19.4.5.4. Context-Free Length Limits19.5. Windows 2000 SNMP DOS19.6. Finding DOS Attacks19.7. SQL-UDP19.8. Conclusion
20. Tracing for Vulnerabilities
20.1. Overview20.1.1. A Vulnerable Program20.1.2. Component Design20.1.2.1. Process Injection20.1.2.2. Machine-Code Analysis20.1.2.2.1. Static Linking20.1.2.2.2. Importing20.1.2.2.3. Inlining20.1.2.3. Function Hooking20.1.2.3.1. Import Hooking20.1.2.3.2. Prelude Hooking20.1.2.3.3. Prologue Hooking20.1.2.4. Data Collection20.1.3. Building VulnTrace20.1.3.1. VTInject20.1.3.2. VulnTrace.dll20.1.4. Using VulnTrace20.1.5. Advanced Techniques20.1.5.1. Fingerprint Systems20.1.5.2. More Vulnerability Classes20.1.5.2.1. Integer Overflows20.1.5.2.2. Format Bugs20.1.5.2.3. Other Classes20.2. Conclusion
21. Binary Auditing: Hacking Closed Source Software
21.1. Binary versus Source-Code Auditing: The Obvious Differences21.2. IDA Pro—The Tool of the Trade21.2.1. Features: A Quick Crash Course21.2.2. Debugging Symbols21.3. Binary Auditing Introduction21.3.1. Stack Frames21.3.1.1. Traditional BP-Based Stack Frames21.3.1.2. Functions without a Frame Pointer21.3.1.3. Non-Traditional BP-Based Stack Frames21.3.2. Calling Conventions21.3.2.1. The C Calling Convention21.3.2.2. The Stdcall Calling Convention21.3.3. Compiler-Generated Code21.3.3.1. Function Layouts21.3.3.2. If Statements21.3.3.3. For and While Loops21.3.3.4. Switch Statements21.3.4. memcpy-Like Code Constructs21.3.5. strlen-Like Code Constructs21.3.6. C++ Code Constructs21.3.7. The this Pointer21.4. Reconstructing Class Definitions21.4.1. vtables21.4.2. Quick but Useful Tidbits21.5. Manual Binary Analysis21.5.1. Quick Examination of Library Calls21.5.2. Suspicious Loops and Write Instructions21.5.3. Higher-Level Understanding and Logic Bugs21.5.4. Graphical Analysis of Binaries21.5.5. Manual Decompilation21.6. Binary Vulnerability Examples21.6.1. Microsoft SQL Server Bugs21.6.2. LSD's RPC-DCOM Vulnerability21.6.3. IIS WebDAV Vulnerability21.7. Conclusion
IV. Advanced Materials
22. Alternative Payload Strategies
22.1. Modifying the Program22.2. The SQL Server 3-Byte Patch22.3. The MySQL 1-Bit Patch22.4. OpenSSH RSA Authentication Patch22.5. Other Runtime Patching Ideas22.5.1. GPG 1.2.2 Randomness Patch22.6. Upload and Run (or Proglet Server)22.7. Syscall Proxies22.8. Problems with Syscall Proxies22.9. Conclusion
23. Writing Exploits that Work in the Wild
23.1. Factors in Unreliability23.1.1. Magic Numbers23.1.2. Versioning23.1.3. Shellcode Problems23.1.3.1. Network Related23.1.3.2. Privilege Related23.1.3.3. Configuration Related23.1.3.4. Host IDS Related23.1.3.5. Thread Related23.2. Countermeasures23.2.1. Preparation23.2.2. Brute Forcing23.2.3. Local Exploits23.2.4. OS/Application Fingerprinting23.2.5. Information Leaks23.3. Conclusion
24. Attacking Database Software
24.1. Network Layer Attacks24.2. Application Layer Attacks24.3. Running Operating System Commands24.3.1. Microsoft SQL Server24.3.2. Oracle24.3.3. IBM DB224.4. Exploiting Overruns at the SQL Level24.4.1. SQL Functions24.4.1.1. Using the CHR/CHAR Function24.5. Conclusion
25. Unix Kernel Overflows
25.1. Kernel Vulnerability Types25.2. 0day Kernel Vulnerabilities25.2.1. OpenBSD exec_ibcs2_coff_prep_zmagic() Stack Overflow25.2.2. The Vulnerability25.3. Solaris vfs_getvfssw() Loadable Kernel Module Traversal Vulnerability25.3.1. The sysfs() System Call25.3.2. The mount() System Call25.4. Conclusion
26. Exploiting Unix Kernel Vulnerabilities
26.1. The exec_ibcs2_coff_prep_zmagic() Vulnerability26.1.1. Calculating Offsets and Breakpoints26.1.2. Overwriting the Return Address and Redirecting Execution26.1.3. Locating the Process Descriptor (or the Proc Structure)26.1.3.1. Stack Lookup26.1.3.2. sysctl() Syscall26.1.4. Kernel Mode Payload Creation26.1.4.1. p_cred and u_cred26.1.4.2. Breaking chroot26.1.5. Returning Back from Kernel Payload26.1.5.1. Return to User Mode: iret Technique26.1.5.2. Return to Kernel Code: sidt Technique and _kernel_text Search26.1.6. Getting root (uid=0)26.2. Solaris vfs_getvfssw() Loadable Kernel Module Path Traversal Exploit26.2.1. Crafting the Exploit26.2.2. The Kernel Module to Load26.2.3. Getting root (uid=0)26.3. Conclusion
27. Hacking the Windows Kernel
27.1. Windows Kernel Mode Flaws—An Increasingly Hunted Species27.2. Introduction to the Windows Kernel27.3. Common Kernel-Mode Programming Flaws27.3.1. Stack Overflows27.3.1.1. The /GS Flag27.3.2. Heap Overflows27.3.3. Insufficient Validation of User-Mode Addresses27.3.4. Repurposing Attacks27.3.5. Shared Object Attacks27.4. Windows System Calls27.4.1. Understanding System Calls27.4.2. Attacking System Calls27.5. Communicating with Device Drivers27.5.1. I/O Control Code Components27.5.2. Finding Flaws in IOCTL Handlers27.5.2.1. Static Analysis27.5.2.2. Trial and Error27.5.2.3. Collect and Fuzz27.6. Kernel-Mode Payloads27.6.1. Elevating a User-Mode Process27.6.2. Running an Arbitrary User-Mode Payload27.6.3. Subverting Kernel Security27.6.4. Installing a Rootkit27.7. Essential Reading for Kernel Shellcoders27.8. Conclusion

Content preview from The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition

Chapter 21. Binary Auditing: Hacking Closed Source Software

Many security-critical and widely deployed code bases are closed source, including some of the dominant operating system families for both servers and desktops. In order to assess the security of closed source software beyond the capabilities of fuzz-testing, binary auditing is a necessity.

In general, binary auditing is considered more difficult than auditing with source code. While this might seem like bad news for beginners, it could also be considered a benefit. There are far fewer people auditing binaries at this point in time, and fewer eyes make for easier work. Many bug classes that are virtually extinct in open source software still linger in closed source commercial code bases.

Binary auditing is still an imperfect science, and many things that can be quite easily verified while auditing source code are conversely quite difficult to determine while examining a binary. With practice and the help of some useful tools, much of the frustration associated with binary auditing can be removed.

Many security researchers do not stretch beyond the limitations of fuzz-testing when auditing commercial software. Although fuzz-testing has proven that it can reveal bugs in software, you really cannot fuzz all possible input patterns to any large piece of software in a reasonable amount of time. Binary auditing can offer a more complete view of the inner workings of an application and security flaws it might contain.

Binary versus ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

The Art of Software Security Assessment: Identifying and Preventing Software Vulnerabilities

Publisher Resources

ISBN: 9780470080238Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition

by Chris Anley, John Heasman, Felix FX Lindner, Gerardo Richarte

Chapter 21. Binary Auditing: Hacking Closed Source Software

Binary versus ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.