book

Reversing: Secrets of Reverse Engineering

Name: Reversing: Secrets of Reverse Engineering
Author: Eldad Eilam
ISBN: 9780764574818

by Eldad Eilam

April 2005

Intermediate to advanced

624 pages

16h 18m

English

Wiley

Read now

Unlock full access

Copyright
Credits
Foreword
Acknowledgments
Introduction
Reverse Engineering and Low-Level SoftwareHow This Book Is OrganizedWho Should Read this BookTools and PlatformsWhat's on the Web SiteWhere to Go from Here?
I. Reversing 101
1. Foundations
1.1. What Is Reverse Engineering?1.2. Software Reverse Engineering: Reversing1.3. Reversing Applications1.3.1. Security-Related Reversing1.3.1.1. Malicious Software1.3.1.2. Reversing Cryptographic Algorithms1.3.1.3. Digital Rights Management1.3.1.4. Auditing Program Binaries1.3.2. Reversing in Software Development1.3.2.1. Achieving Interoperability with Proprietary Software1.3.2.2. Developing Competing Software1.3.2.3. Evaluating Software Quality and Robustness1.4. Low-Level Software1.4.1. Assembly Language1.4.2. Compilers1.4.3. Virtual Machines and Bytecodes1.4.4. Operating Systems1.5. The Reversing Process1.5.1. System-Level Reversing1.5.2. Code-Level Reversing1.6. The Tools1.6.1. System-Monitoring Tools1.6.2. Disassemblers1.6.3. Debuggers1.6.4. Decompilers1.7. Is Reversing Legal?1.7.1. Interoperability1.7.2. Competition1.7.3. Copyright Law1.7.4. Trade Secrets and Patents1.7.5. The Digital Millenium Copyright Act1.7.6. DMCA Cases1.7.7. License Agreement Considerations1.8. Code Samples & Tools1.9. Conclusion
2. Low-Level Software
2.1. High-Level Perspectives2.1.1. Program Structure2.1.1.1. Modules2.1.1.2. Common Code Constructs2.1.2. Data Management2.1.2.1. Variables2.1.2.2. User-Defined Data Structures2.1.2.3. Lists2.1.3. Control Flow2.1.4. High-Level Languages2.1.4.1. C2.1.4.2. C++2.1.4.3. Java2.1.4.4. C#2.2. Low-Level Perspectives2.2.1. Low-Level Data Management2.2.1.1. Registers2.2.1.2. The Stack2.2.1.3. Heaps2.2.1.4. Executable Data Sections2.2.2. Control Flow2.3. Assembly Language 1012.3.1. Registers2.3.2. Flags2.3.3. Instruction Format2.3.4. Basic Instructions2.3.4.1. Moving Data2.3.4.2. Arithmetic2.3.4.3. Comparing Operands2.3.4.4. Conditional Branches2.3.4.5. Function Calls2.3.5. Examples2.4. A Primer on Compilers and Compilation2.4.1. Defining a Compiler2.4.2. Compiler Architecture2.4.2.1. Front End2.4.2.2. Intermediate Representations2.4.2.3. Optimizer2.4.2.3.1. Code Structure2.4.2.3.2. Redundancy Elimination2.4.2.4. Back End2.4.3. Listing Files2.4.4. Specific Compilers2.5. Execution Environments2.5.1. Software Execution Environments (Virtual Machines)2.5.1.1. Bytecodes2.5.1.2. Interpreters2.5.1.3. Just-in-Time Compilers2.5.1.4. Reversing Strategies2.5.2. Hardware Execution Environments in Modern Processors2.5.2.1. Intel NetBurst2.5.2.2. μops (Micro-Ops)2.5.2.3. Pipelines2.5.2.4. Branch Prediction2.6. Conclusion
3. Windows Fundamentals
3.1. Components and Basic Architecture3.1.1. Brief History3.1.2. Features3.1.3. Supported Hardware3.2. Memory Management3.2.1. Virtual Memory and Paging3.2.1.1. Paging3.2.1.2. Page Faults3.2.2. Working Sets3.2.3. Kernel Memory and User Memory3.2.4. The Kernel Memory Space3.2.5. Section Objects3.2.6. VAD Trees3.2.7. User-Mode Allocations3.2.8. Memory Management APIs3.3. Objects and Handles3.3.1. Named objects3.4. Processes and Threads3.4.1. Processes3.4.2. Threads3.4.3. Context Switching3.4.4. Synchronization Objects3.4.5. Process Initialization Sequence3.5. Application Programming Interfaces3.5.1. The Win32 API3.5.2. The Native API3.5.3. System Calling Mechanism3.6. Executable Formats3.6.1. Basic Concepts3.6.2. Image Sections3.6.3. Section Alignment3.6.4. Dynamically Linked Libraries3.6.5. Headers3.6.6. Imports and Exports3.6.7. Directories3.7. Input and Output3.7.1. The I/O System3.7.2. The Win32 Subsystem3.7.2.1. Object Management3.8. Structured Exception Handling3.9. Conclusion
4. Reversing Tools
4.1. Different Reversing Approaches4.1.1. Offline Code Analysis (Dead-Listing)4.1.2. Live Code Analysis4.1.3. Disassemblers4.1.3.1. IDA Pro4.1.4. ILDasm4.2. Debuggers4.2.1. User-Mode Debuggers4.2.1.1. OllyDbg4.2.1.2. User Debugging in WinDbg4.2.1.3. IDA Pro4.2.1.4. PEBrowse Professional Interactive4.2.2. Kernel-Mode Debuggers4.2.2.1. Kernel Debugging in WinDbg4.2.2.2. Numega SoftICE4.2.2.3. Kernel Debugging on Virtual Machines4.3. Decompilers4.4. System-Monitoring Tools4.5. Patching Tools4.5.1. Hex Workshop4.6. Miscellaneous Reversing Tools4.6.1. Executable-Dumping Tools4.6.1.1. DUMPBIN4.6.1.2. PEView4.6.1.3. PEBrowse Professional4.7. Conclusion

II. Applied Reversing
5. Beyond the Documentation
5.1. Reversing and Interoperability5.2. Laying the Ground Rules5.3. Locating Undocumented APIs5.3.1. What Are We Looking For?5.4. Case Study: The Generic Table API in NTDLL.DLL5.4.1. RtlInitializeGenericTable5.4.2. RtlNumberGenericTableElements5.4.3. RtlIsGenericTableEmpty5.4.4. RtlGetElementGenericTable5.4.4.1. Setup and Initialization5.4.4.2. Logic and Structure5.4.4.3. Search Loop 15.4.4.4. Search Loop 25.4.4.5. Search Loop 35.4.4.6. Search Loop 45.4.4.7. Reconstructing the Source Code5.4.5. RtlInsertElementGenericTable5.4.5.1. RtlLocateNodeGenericTable5.4.5.1.1. The Callback5.4.5.1.2. High-Level Theories5.4.5.1.3. Callback Parameters5.4.5.1.4. Summarizing the Findings5.4.5.2. RtlRealInsertElementWorker5.4.5.2.1. Linking the Element5.4.5.2.2. Copying the Element5.4.5.2.3. Splaying the Table5.4.5.3. Splay Trees5.4.6. RtlLookupElementGenericTable5.4.7. RtlDeleteElementGenericTable5.4.8. Putting the Pieces Together5.5. Conclusion
6. Deciphering File Formats
6.1. Cryptex6.2. Using Cryptex6.3. Reversing Cryptex6.4. The Password Verification Process6.4.1. Catching the "Bad Password" Message6.4.2. The Password Transformation Algorithm6.4.3. Hashing the Password6.5. The Directory Layout6.5.1. Analyzing the Directory Processing Code6.5.2. Analyzing a File Entry6.6. Dumping the Directory Layout6.7. The File Extraction Process6.8. Scanning the File List6.8.1. Decrypting the File6.8.2. The Floating-Point Sequence6.8.3. The Decryption Loop6.8.4. Verifying the Hash Value6.9. The Big Picture6.10. Digging Deeper6.11. Conclusion
7. Auditing Program Binaries
7.1. Defining the Problem7.2. Vulnerabilities7.2.1. Stack Overflows7.2.1.1. A Simple Stack Vulnerability7.2.1.2. Intrinsic Implementations7.2.1.3. Stack Checking7.2.1.4. Nonexecutable Memory7.2.2. Heap Overflows7.2.3. String Filters7.2.4. Integer Overflows7.2.4.1. Arithmetic Operations on User-Supplied Integers7.2.5. Type Conversion Errors7.3. Case-Study: The IIS Indexing Service Vulnerability7.3.1. CVariableSet::AddExtensionControlBlock7.3.2. DecodeURLEscapes7.4. Conclusion
8. Reversing Malware
8.1. Types of Malware8.1.1. Viruses8.1.2. Worms8.1.3. Trojan Horses8.1.4. Backdoors8.1.5. Mobile Code8.1.6. Adware/Spyware8.2. Sticky Software8.3. Future Malware8.3.1. Information-Stealing Worms8.3.2. BIOS/Firmware Malware8.4. Uses of Malware8.5. Malware Vulnerability8.6. Polymorphism8.7. Metamorphism8.8. Establishing a Secure Environment8.9. The HackArmy Backdoor8.9.1. Unpacking the Executable8.9.2. Initial Impressions8.9.3. The Initial Installation8.9.4. Initializing Communications8.9.5. Connecting to the Server8.9.6. Joining the Channel8.9.7. Communicating with the Backdoor8.9.8. Running SOCKS4 Servers8.9.9. Clearing the Crime Scene8.10. The Hackarmy Backdoor: A Command Reference8.11. Conclusion
III. Cracking
9. Piracy and Copy Protection
9.1. Copyrights in the New World9.2. The Social Aspect9.3. Software Piracy9.3.1. Defining the Problem9.3.2. Class Breaks9.3.3. Requirements9.3.4. The Theoretically Uncrackable Model9.4. Types of Protection9.4.1. Media-Based Protections9.4.2. Serial Numbers9.4.3. Challenge Response and Online Activations9.4.4. Hardware-Based Protections9.4.5. Software as a Service9.5. Advanced Protection Concepts9.5.1. Crypto-Processors9.6. Digital Rights Management9.6.1. DRM Models9.6.1.1. The Windows Media Rights Manager9.6.1.2. Secure Audio Path9.7. Watermarking9.8. Trusted Computing9.9. Attacking Copy Protection Technologies9.10. Conclusion
10. Antireversing Techniques
10.1. Why Antireversing?10.2. Basic Approaches to Antireversing10.3. Eliminating Symbolic Information10.4. Code Encryption10.5. Active Antidebugger Techniques10.5.1. Debugger Basics10.5.2. The IsDebuggerPresent API10.5.3. SystemKernelDebuggerInformation10.5.4. Detecting SoftICE Using the Single-Step Interrupt10.5.5. The Trap Flag10.5.6. Code Checksums10.6. Confusing Disassemblers10.6.1. Linear Sweep Disassemblers10.6.2. Recursive Traversal Disassemblers10.6.3. Applications10.7. Code Obfuscation10.8. Control Flow Transformations10.8.1. Opaque Predicates10.8.2. Confusing Decompilers10.8.3. Table Interpretation10.8.4. Inlining and Outlining10.8.5. Interleaving Code10.8.6. Ordering Transformations10.9. Data Transformations10.9.1. Modifying Variable Encoding10.9.2. Restructuring Arrays10.10. Conclusion
11. Breaking Protections
11.1. Patching11.2. Keygenning11.3. Ripping Key-Generation Algorithms11.4. Advanced Cracking: Defender11.4.1. Reversing Defender's Initialization Routine11.4.2. Analyzing the Decrypted Code11.4.3. SoftICE's Disappearance11.4.4. Reversing the Secondary Thread11.4.5. Defeating the "Killer" Thread11.4.6. Loading KERNEL32.DLL11.4.7. Reencrypting the Function11.4.8. Back at the Entry Point11.4.9. Parsing the Program Parameters11.4.10. Processing the Username11.4.11. Validating User Information11.4.12. Unlocking the Code11.4.13. Brute-Forcing Your Way through Defender11.5. Protection Technologies in Defender11.5.1. Localized Function-Level Encryption11.5.1.1. Relatively Strong Cipher Block Chaining11.5.1.2. Reencrypting11.5.2. Obfuscated Application/Operating System Interface11.5.3. Processor Time-Stamp Verification Thread11.5.4. Runtime Generation of Decryption Keys11.5.4.1. Interdependent Keys11.5.4.2. User-Input-Based Decryption Keys11.5.5. Heavy Inlining11.6. Conclusion
IV. Beyond Disassembly
12. Reversing .NET
12.1. Ground Rules12.2. .NET Basics12.2.1. Managed Code12.2.2. .NET Programming Languages12.2.3. Common Type System (CTS)12.3. Intermediate Language (IL)12.3.1. The Evaluation Stack12.3.2. Activation Records12.3.3. IL Instructions12.3.4. IL Code Samples12.3.4.1. Counting Items12.3.4.2. A Linked List Sample12.3.4.2.1. The ListItem Class12.3.4.2.2. The LinkedList Class12.3.4.2.3. The StringItem Class12.4. Decompilers12.5. Obfuscators12.5.1. Renaming Symbols12.5.2. Control Flow Obfuscation12.5.3. Breaking Decompilation and Disassembly12.6. Reversing Obfuscated Code12.6.1. XenoCode Obfuscator12.6.2. DotFuscator by Preemptive Solutions12.6.3. Remotesoft Obfuscator and Linker12.6.4. Remotesoft Protector12.6.5. Precompiled Assemblies12.6.6. Encrypted Assemblies12.7. Conclusion
13. Decompilation
13.1. Native Code Decompilation: An Unsolvable Problem?13.2. Typical Decompiler Architecture13.3. Intermediate Representations13.3.1. Expressions and Expression Trees13.3.2. Control Flow Graphs13.4. The Front End13.4.1. Semantic Analysis13.4.2. Generating Control Flow Graphs13.5. Code Analysis13.5.1. Data-Flow Analysis13.5.1.1. Single Static Assignment (SSA)13.5.1.2. Data Propagation13.5.1.3. Register Variable Identification13.5.1.4. Data Type Propagation13.5.2. Type Analysis13.5.2.1. Primitive Data Types13.5.2.2. Complex Data Types13.5.3. Control Flow Analysis13.5.4. Finding Library Functions13.6. The Back End13.7. Real-World IA-32 Decompilation13.8. Conclusion
A. Deciphering Code Structures
A.1. Understanding Low-Level LogicA.1.1. Comparing OperandsA.1.1.1. Signed ComparisonsA.1.1.2. Unsigned ComparisonsA.1.2. The Conditional CodesA.1.2.1. Signed Conditional CodesA.1.2.2. Unsigned Conditional CodesA.2. Control Flow & Program LayoutA.2.1. Deciphering FunctionsA.2.1.1. Internal FunctionsA.2.1.2. Imported FunctionsA.2.2. Single-Branch ConditionalsA.2.3. Two-Way ConditionalsA.2.4. Multiple-Alternative ConditionalsA.2.5. Compound ConditionalsA.2.5.1. Logical OperatorsA.2.5.2. Simple CombinationsA.2.5.3. Complex CombinationsA.2.6. n-way Conditional (Switch Blocks)A.2.6.1. Table ImplementationA.2.6.2. Tree ImplementationA.2.7. LoopsA.2.7.1. Pretested LoopsA.2.7.2. Posttested LoopsA.2.7.3. Loop Break ConditionsA.2.7.4. Loop Skip-Cycle StatementsA.2.7.5. Loop UnrollingA.2.8. Branchless LogicA.2.8.1. Pure Arithmetic ImplementationsA.2.8.2. Predicated ExecutionA.2.8.2.1. Set Byte on Condition (SETcc)A.2.8.2.2. Conditional Move (CMOVcc)A.3. Effects of Working-Set Tuning on ReversingA.3.1. Function-Level Working-Set TuningA.3.2. Line-Level Working-Set Tuning
B. Understanding Compiled Arithmetic
B.1. Arithmetic FlagsB.1.1. The Overflow Flags (CF and OF)B.1.2. The Zero Flag (ZF)B.1.3. The Sign Flag (SF)B.1.4. The Parity Flag (PF)B.2. Basic Integer ArithmeticB.2.1. Addition and SubtractionB.2.2. Multiplication and DivisionB.2.2.1. MultiplicationB.2.2.2. DivisionB.2.2.2.1. Understanding Reciprocal-MultiplicationsB.2.2.2.2. Deciphering Reciprocal-MultiplicationsB.2.3. ModuloB.3. 64-Bit ArithmeticB.3.1. AdditionB.3.2. SubtractionB.3.3. MultiplicationB.3.4. DivisionB.4. Type ConversionsB.4.1. Zero ExtendingB.4.2. Sign ExtendingB.4.2.1. To 32 BitsB.4.2.2. To 64 Bits
C. Deciphering Program Data
C.1. The StackC.1.1. Stack FramesC.1.2. The ENTER and LEAVE InstructionsC.1.3. Calling ConventionsC.1.3.1. The cdecl Calling ConventionC.1.3.2. The fastcall Calling ConventionC.1.3.3. The stdcall Calling ConventionC.1.3.4. The C++ Class Member Calling Convention (thiscall)C.2. Basic Data ConstructsC.2.1. Global VariablesC.2.2. Local VariablesC.2.2.1. Stack-BasedC.2.2.1.1. Overwriting Passed ParametersC.2.2.2. Register-BasedC.2.3. Imported VariablesC.2.4. ConstantsC.2.5. Thread-Local Storage (TLS)C.3. Data StructuresC.3.1. Generic Data StructuresC.3.1.1. AlignmentC.3.2. ArraysC.3.2.1. Generic Data Type ArraysC.3.2.2. Data Structure ArraysC.3.3. Linked ListsC.3.3.1. Singly Linked ListsC.3.3.2. Doubly Linked ListsC.3.4. TreesC.4. ClassesC.4.1. Data MembersC.4.2. Data Members in Inherited ClassesC.4.3. Class MethodsC.4.4. Virtual FunctionsC.4.5. Identifying Virtual Function CallsC.4.6. Identifying Constructors of Objects with Inheritance
D. Citations

Content preview from Reversing: Secrets of Reverse Engineering

Credits

Executive Editor

Robert Elliott

Development Editor

Eileen Bien Calabro

Copy Editor

Foxxe Editorial Services

Editorial Manager

Mary Beth Wakefield

Vice President & Executive Group Publisher

Richard Swadley

Vice President and Publisher

Joseph B. Wikert

Project Editor

Pamela Hanley

Project Coordinator

Ryan Steffen

Graphics and Production Specialists

Denny Hager

Jennifer Heleine

Lynsey Osborn

Mary Gillot Virgin

Quality Control Technician

Leeann Harney

Proofreading and Indexing

TECHBOOKS Production Services

Cover Designer

Michael Trent

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Reverse Engineering and Exploit Development

Publisher Resources

ISBN: 9780764574818Purchase book

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Reversing: Secrets of Reverse Engineering

by Eldad Eilam

Credits

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.