book

The Shellcoder's Handbook: Discovering and Exploiting Security Holes

Name: The Shellcoder's Handbook: Discovering and Exploiting Security Holes
ISBN: 9780764544682

by Jack Koziol, Dave Aitel, David Litchfield, Chris Anley, Sinan noir Eren, Neel Mehta, Riley Hassell

April 2004

Intermediate to advanced

644 pages

13h 49m

English

Wiley

Read now

Unlock full access

Copyright
About the Authors
Credits
Acknowledgments
I. Introduction to Exploitation: Linux on x86
1. Before You Begin
1.1. Basic Concepts1.1.1. Memory Management1.1.2. Assembly1.1.3. Registers1.2. Recognizing C++ Code Constructs in Assembly1.3. Conclusion
2. Stack Overflows
2.1. Buffers2.2. The Stack2.2.1. Functions and the Stack2.3. Overflowing Buffers on the Stack2.3.1. Controlling EIP2.4. Using an Exploit to Get Root Privileges2.4.1. The Address Problem2.4.2. The NOP Method2.5. Defeating a Non-Executable Stack2.5.1. Return to libc2.6. Conclusion
3. Shellcode
3.1. Understanding System Calls3.2. Writing Shellcode for the exit() Syscall3.3. Injectable Shellcode3.4. Spawning a Shell3.5. Conclusion
4. Introduction to Format String Bugs
4.1. Prerequisites4.2. What Is a Format String?4.3. What Is a Format String Bug?4.4. Format String Exploits4.4.1. Crashing Services4.4.2. Information Leakage4.5. Controlling Execution for Exploitation4.6. Why Did This Happen?4.7. Format String Technique Roundup4.8. Conclusion
5. Introduction to Heap Overflows
5.1. What Is a Heap?5.1.1. How a Heap Works5.2. Finding Heap Overflows5.2.1. Basic Heap Overflows5.2.2. Intermediate Heap Overflows5.2.3. Advanced Heap Overflow Exploitation5.2.3.1. What to Overwrite5.2.3.1.1. GOT Entries5.2.3.1.2. Global Function Pointers5.2.3.1.3. DTORS5.2.3.1.4. atexit Handlers5.2.3.1.5. Stack Values5.3. Conclusion

II. Exploiting More Platforms: Windows, Solaris, and Tru64
6. The Wild World of Windows
6.1. How Does Windows Differ from Linux?6.1.1. Win32 API and PE-COFF6.2. Heaps6.2.1. Threading6.3. The Genius and Idiocy of the Distributed Common Object Model and DCE-RPC6.3.1. Recon6.3.2. Exploitation6.3.3. Tokens and Impersonation6.3.4. Exception Handling under Win326.4. Debugging Windows6.4.1. Bugs in Win326.4.2. Writing Windows Shellcode6.4.3. A Hacker's Guide to the Win32 API6.4.4. A Windows Family Tree from the Hacker's Perspective6.5. Conclusion
7. Windows Shellcode
7.1. Syntax and Filters7.2. Setting Up7.3. Parsing the PEB7.3.1. Heapoverflow.c Analysis7.4. Searching with Windows Exception Handling7.5. Popping a Shell7.5.1. Why You Should Never Pop a Shell on Windows7.6. Conclusion
8. Windows Overflows
8.1. Stack-Based Buffer Overflows8.1.1. Frame-Based Exception Handlers8.1.2. Abusing Frame-Based Exception Handling on Windows 2003 Server8.1.2.1. Abusing an Existing Handler8.1.2.2. Find a block of code in an address not associated with a module that will get us back to our buffer8.1.2.3. Find a block of code in the address space of a module that does not have a Load Configuration Directory8.1.3. A Final Note about Frame-Based Handler Overwrites8.2. Stack Protection and Windows 2003 Server8.3. Heap-Based Buffer Overflows8.3.1. The Process Heap8.3.2. Dynamic Heaps8.3.3. Working with the Heap8.3.4. How the Heap Works8.4. Exploiting Heap-Based Overflows8.4.1. Overwrite Pointer to RtlEnterCriticalSection in the PEB8.4.2. Overwrite Pointer to First Vectored Handler at 77FC32108.4.3. Overwrite Pointer to Unhandled Exception Filter8.4.4. Overwrite Pointer to Exception Handler in Thread Environment Block8.4.5. Repairing the Heap8.4.6. Other Aspects of Heap-Based Overflows8.4.6.1. COM Objects and the Heap8.4.6.2. Overflowing Logic Program Control Data8.4.7. Wrapping Up the Heap8.5. Other Overflows8.5.1. .data section overflows8.5.2. TEB/PEB Overflows8.6. Exploiting Buffer Overflows and Non-Executable Stacks8.7. Conclusion
9. Overcoming Filters
9.1. Writing Exploits for Use with an Alphanumeric Filter9.2. Writing Exploits for Use with a Unicode Filter9.2.1. What Is Unicode?9.2.2. Converting from ASCII to Unicode9.3. Exploiting Unicode-Based Vulnerabilities9.3.1. The Available Instruction Set in Unicode Exploits9.4. The Venetian Method9.4.1. An ASCII Venetian Implementation9.5. Decoder and Decoding9.5.1. The Decoder Code9.5.2. Getting a Fix on the Buffer Address9.6. Conclusion
10. Introduction to Solaris Exploitation
10.1. Introduction to the SPARC Architecture10.1.1. Registers and Register Windows10.1.2. The Delay Slot10.1.3. Synthetic Instructions10.2. Solaris/SPARC Shellcode Basics10.2.1. Self-Location Determination and SPARC Shellcode10.2.2. Simple SPARC exec Shellcode10.2.3. Useful System Calls on Solaris10.2.4. NOP and Padding Instructions10.3. Solaris/SPARC Stack Frame Introduction10.4. Stack-Based Overflow Methodologies10.4.1. Arbitrary Size Overflow10.4.2. Register Windows and Stack Overflow Complications10.4.3. Other Complicating Factors10.4.4. Possible Solutions10.4.5. Off-By-One Stack Overflow Vulnerabilities10.4.6. Shellcode Locations10.5. Stack Overflow Exploitation In Action10.5.1. The Vulnerable Program10.5.2. The Exploit10.6. Heap-Based Overflows on Solaris/SPARC10.6.1. Solaris System V Heap Introduction10.6.2. Heap Tree Structure10.7. Basic Exploit Methodology (t_delete)10.7.1. Standard Heap Overflow Limitations10.7.2. Targets for Overwrite10.7.2.1. The Bottom Chunk10.7.2.2. Small Chunk Corruption10.8. Other Heap-Related Vulnerabilities10.8.1. Off-by-One Overflows10.8.2. Double Free Vulnerabilities10.8.3. Arbitrary Free Vulnerabilities10.9. Heap Overflow Example10.9.1. The Vulnerable Program10.10. Other Solaris Exploitation Techniques10.10.1. Static Data Overflows10.10.2. Bypassing the Non-Executable Stack Protection10.11. Conclusion
11. Advanced Solaris Exploitation
11.1. Single Stepping the Dynamic Linker11.2. Various Style Tricks for Solaris SPARC Heap Overflows11.3. Advanced Solaris/SPARC Shellcode11.4. Conclusion
12. HP Tru64 Unix Exploitation
12.1. The Alpha Architecture12.1.1. Alpha Registers12.1.2. Instruction Set12.1.3. Calling Conventions12.2. Retrieving the Program Counter (GetPC)12.3. System Call Invocation12.4. XOR Decoder for Shellcode12.5. .end main setuid + execve Shellcode12.5.1. Code the setuid(0) + execve("/bin/sh", . . .) systemcalls12.5.2. Compile the Assembly Code and Extract the Main Function12.5.3. Encode the Extracted opcodes with the XOR Key12.5.4. Plug the Encoded Code into the XOR Decoder12.5.5. Compile and Extract the Final Shellcode12.6. Connect-Back Shellcode12.7. Find-Socket Shellcode12.8. Bind-Socket Shellcode12.9. Stack Overflow Exploitation12.9.1. Defeating the Non-Executable Stack12.10. Exploiting rpc.ttdbserver12.11. Conclusion
III. Vulnerability Discovery
13. Establishing a Working Environment
13.1. What You Need for Reference13.2. What You Need for Code13.2.1. gcc13.2.2. gdb13.2.3. NASM13.2.4. WinDbg13.2.5. OllyDbg13.2.6. SoftICE13.2.7. Visual C++13.2.8. Python13.3. What You Need for Investigation13.3.1. Useful Custom Scripts/Tools13.3.1.1. An Offset Finder13.3.1.2. Generic Fuzzers13.3.1.3. The Debug Trick13.3.2. All Platforms13.3.3. Unix13.3.3.1. ltrace13.3.3.2. strace13.3.3.3. fstat (BSD)13.3.3.4. tcpdump13.3.3.5. Ethereal13.3.4. Windows13.3.4.1. IDA Pro Disassembler13.4. What You Need to Know13.4.1. Paper Archives13.5. Optimizing Shellcode Development13.5.1. Plan the Exploit13.5.2. Write the Shellcode in Inline Assembler13.5.3. Maintain a Shellcode Library13.5.4. Make It Continue Nicely13.5.5. Make the Exploit Stable13.5.6. Make It Steal the Connection13.6. Conclusion
14. Fault Injection
14.1. Design Overview14.1.1. Input Generation14.1.1.1. Manual Generation14.1.1.2. Automated Generation14.1.1.3. Live Capture14.1.1.4. "Fuzz" Generation14.1.2. Fault Injection14.1.3. Modification Engines14.1.3.1. Delimiting Logic14.1.3.2. Getting around Input Sanitization14.1.4. Fault Delivery14.1.5. Nagel Algorithm14.1.6. Timing14.1.7. Heuristics14.1.8. Stateless versus State-Based Protocols14.2. Fault Monitoring14.2.1. Using a Debugger14.2.2. FaultMon14.3. Putting It Together14.4. Conclusion
15. The Art of Fuzzing
15.1. General Theory of Fuzzing15.1.1. Static Analysis versus Fuzzing15.1.2. Fuzzing Is Scalable15.2. Weaknesses in Fuzzers15.3. Modeling Arbitrary Network Protocols15.4. Other Fuzzer Possibilities15.4.1. Bit Flipping15.4.2. Modifying Open Source Programs15.4.3. Fuzzing with Dynamic Analysis15.5. SPIKE15.5.1. What Is a Spike?15.5.2. Why Use the SPIKE Data Structure to Model Network Protocols?15.5.2.1. Various Programs Included with SPIKE15.5.2.2. SPIKE Example: dtlogin15.6. Other Fuzzers15.7. Conclusion
16. Source Code Auditing: Finding Vulnerabilities in C-Based Languages
16.1. Tools16.1.1. Cscope16.1.2. Ctags16.1.3. Editors16.1.4. Cbrowser16.2. Automated Source Code Analysis Tools16.3. Methodology16.3.1. Top-Down (Specific) Approach16.3.2. Bottom-Up Approach16.3.3. Selective Approach16.4. Vulnerability Classes16.4.1. Generic Logic Errors16.4.2. (Almost) Extinct Bug Classes16.4.3. Format Strings16.4.4. Generic Incorrect Bounds-Checking16.4.5. Loop Constructs16.4.6. Off-by-One Vulnerabilities16.4.7. Non-Null Termination Issues16.4.8. Skipping Null-Termination Issues16.4.9. Signed Comparison Vulnerabilities16.4.10. Integer-Related Vulnerabilities16.4.11. Different-Sized Integer Conversions16.4.12. Double Free Vulnerabilities16.4.13. Out-of-Scope Memory Usage Vulnerabilities16.4.14. Uninitialized Variable Usage16.4.15. Use After Free Vulnerabilities16.4.16. Multithreaded Issues and Re-Entrant Safe Code16.5. Beyond Recognition: A Real Vulnerability versus a Bug16.6. Conclusion
17. Instrumented Investigation: A Manual Approach
17.1. Philosophy17.2. Oracle extproc Overflow17.3. Common Architectural Failures17.3.1. Problems Happen at Boundaries17.3.1.1. A Process Calling into an External Process on the Same Host17.3.1.2. A Process Calling into an External, Dynamically Loaded Library17.3.1.3. A Process Calling into a Function on a Remote Host17.3.2. Problems Happen When Data Is Translated17.3.3. Problems Cluster in Areas of Asymmetry17.3.4. Problems Occur When Authentication and Authorization Are Confused17.3.5. Problems Occur in the Dumbest Places17.4. Bypassing Input Validation and Attack Detection17.4.1. Stripping Bad Data17.4.2. Using Alternate Encodings17.4.3. Using File-Handling Features17.4.3.1. Required String Is Present in Path17.4.3.2. Prohibited String Not Present in Path17.4.3.3. Incorrect Behavior Based on File Extension17.4.4. Evading Attack Signatures17.4.5. Defeating Length Limitations17.4.5.1. Sea Monkey Data17.4.5.2. Harmful Truncation—Severing Escape Characters17.4.5.3. Multiple Attempts17.4.5.4. Context-Free Length Limits17.5. Windows 2000 SNMP DOS17.6. Finding DOS Attacks17.7. SQL-UDP17.8. Conclusion
18. Tracing for Vulnerabilities
18.1. Overview18.1.1. A Vulnerable Program18.1.2. Component Design18.1.2.1. Process Injection18.1.2.2. Machine-Code Analysis18.1.2.2.1. Static Linking18.1.2.2.2. Importing18.1.2.2.3. Inlining18.1.2.3. Function Hooking18.1.2.3.1. Import Hooking18.1.2.3.2. Prelude Hooking18.1.2.3.3. Prologue Hooking18.1.2.4. Data Collection18.1.3. Building VulnTrace18.1.3.1. VTInject18.1.3.2. VulnTrace.dll18.1.4. Using VulnTrace18.1.5. Advanced Techniques18.1.5.1. Fingerprint Systems18.1.5.2. More Vulnerability Classes18.1.5.2.1. Integer Overflows18.1.5.2.2. Format Bugs18.1.5.2.3. Other Classes18.2. Conclusion
19. Binary Auditing: Hacking Closed Source Software
19.1. Binary versus Source-Code Auditing: The Obvious Differences19.2. IDA Pro—The Tool of the Trade19.2.1. Features: A Quick Crash Course19.2.2. Debugging Symbols19.3. Binary Auditing Introduction19.3.1. Stack Frames19.3.1.1. Traditional BP-Based Stack Frames19.3.1.2. Functions without a Frame Pointer19.3.1.3. Non-Traditional BP-Based Stack Frames19.3.2. Calling Conventions19.3.2.1. The C Calling Convention19.3.2.2. The Stdcall Calling Convention19.3.3. Compiler-Generated Code19.3.3.1. Function Layouts19.3.3.2. If Statements19.3.3.3. For and While Loops19.3.3.4. Switch Statements19.3.4. memcpy-Like Code Constructs19.3.5. strlen-Like Code Constructs19.3.6. C++ Code Constructs19.3.7. The this Pointer19.4. Reconstructing Class Definitions19.4.1. vtables19.4.2. Quick but Useful Tidbits19.5. Manual Binary Analysis19.5.1. Quick Examination of Library Calls19.5.2. Suspicious Loops and Write Instructions19.5.3. Higher-Level Understanding and Logic Bugs19.5.4. Graphical Analysis of Binaries19.5.5. Manual Decompilation19.6. Binary Vulnerability Examples19.6.1. Microsoft SQL Server Bugs19.6.2. LSD's RPC-DCOM Vulnerability19.6.3. IIS WebDAV Vulnerability19.7. Conclusion
IV. Advanced Materials
20. Alternative Payload Strategies
20.1. Modifying the Program20.2. The SQL Server 3-Byte Patch20.3. The MySQL 1-Bit Patch20.4. OpenSSH RSA Authentication Patch20.5. Other Runtime Patching Ideas20.5.1. GPG 1.2.2 Randomness Patch20.6. Upload and Run (or Proglet Server)20.7. Syscall Proxies20.8. Problems with Syscall Proxies20.9. Conclusion
21. Writing Exploits that Work in the Wild
21.1. Factors in Unreliability21.1.1. Magic Numbers21.1.2. Versioning21.1.3. Shellcode Problems21.1.3.1. Network Related21.1.3.2. Privilege Related21.1.3.3. Configuration Related21.1.3.4. Host IDS Related21.1.3.5. Thread Related21.2. Countermeasures21.2.1. Preparation21.2.2. Brute Forcing21.2.3. Local Exploits21.2.4. OS/Application Fingerprinting21.2.5. Information Leaks21.3. Conclusion
22. Attacking Database Software
22.1. Network Layer Attacks22.2. Application Layer Attacks22.3. Running Operating System Commands22.3.1. Microsoft SQL Server22.3.2. Oracle22.3.3. IBM DB222.4. Exploiting Overruns at the SQL Level22.4.1. SQL Functions22.4.1.1. Using the CHR/CHAR Function22.5. Conclusion
23. Kernel Overflows
23.1. Kernel Vulnerability Types23.2. 0day Kernel Vulnerabilities23.2.1. OpenBSD exec_ibcs2_coff_prep_zmagic() Stack Overflow23.2.2. The Vulnerability23.3. Solaris vfs_getvfssw() Loadable Kernel Module Traversal Vulnerability23.3.1. The sysfs() System Call23.3.2. The mount() System Call23.4. Conclusion
24. Exploiting Kernel Vulnerabilities
24.1. The exec_ibcs2_coff_prep_zmagic() Vulnerability24.1.1. Calculating Offsets and Breakpoints24.1.2. Overwriting the Return Address and Redirecting Execution24.1.3. Locating the Process Descriptor (or the Proc Structure)24.1.3.1. Stack Lookup24.1.3.2. sysctl() Syscall24.1.4. Kernel Mode Payload Creation24.1.4.1. p_cred and u_cred24.1.4.2. Breaking chroot24.1.5. Returning Back from Kernel Payload24.1.5.1. Return to User Mode: iret Technique24.1.5.2. Return to Kernel Code: sidt Technique and _kernel_text Search24.1.6. Getting root (uid=0)24.2. Solaris vfs_getvfssw() Loadable Kernel Module Path Traversal Exploit24.2.1. Crafting the Exploit24.2.2. The Kernel Module to Load24.2.3. Getting root (uid=0)24.3. Conclusion

Overview

Examines where security holes come from, how to discover them, how hackers exploit them and take control of systems on a daily basis, and most importantly, how to close these security holes so they never occur again
A unique author team-a blend of industry and underground experts- explain the techniques that readers can use to uncover security holes in any software or operating system
Shows how to pinpoint vulnerabilities in popular operating systems (including Windows, Linux, and Solaris) and applications (including MS SQL Server and Oracle databases)
Details how to deal with discovered vulnerabilities, sharing some previously unpublished advanced exploits and techniques

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition

Publisher Resources

ISBN: 9780764544682Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills