Mastering Malware Analysis

Mastering Malware Analysis

by Alexey Kleymenov, Amr Thabet
Released June 2019
Publisher(s): Packt Publishing
ISBN: 9781789610789

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Master malware analysis to protect your systems from getting infected

Key Features

  • Set up and model solutions, investigate malware, and prevent it from occurring in future
  • Learn core concepts of dynamic malware analysis, memory forensics, decryption, and much more
  • A practical guide to developing innovative solutions to numerous malware incidents

Book Description

With the ever-growing proliferation of technology, the risk of encountering malicious code or malware has also increased. Malware analysis has become one of the most trending topics in businesses in recent years due to multiple prominent ransomware attacks. Mastering Malware Analysis explains the universal patterns behind different malicious software types and how to analyze them using a variety of approaches. You will learn how to examine malware code and determine the damage it can possibly cause to your systems to ensure that it won't propagate any further. Moving forward, you will cover all aspects of malware analysis for the Windows platform in detail. Next, you will get to grips with obfuscation and anti-disassembly, anti-debugging, as well as anti-virtual machine techniques. This book will help you deal with modern cross-platform malware. Throughout the course of this book, you will explore real-world examples of static and dynamic malware analysis, unpacking and decrypting, and rootkit detection. Finally, this book will help you strengthen your defenses and prevent malware breaches for IoT devices and mobile platforms. By the end of this book, you will have learned to effectively analyze, investigate, and build innovative solutions to handle any malware incidents.

What you will learn

  • Explore widely used assembly languages to strengthen your reverse-engineering skills
  • Master different executable file formats, programming languages, and relevant APIs used by attackers
  • Perform static and dynamic analysis for multiple platforms and file types
  • Get to grips with handling sophisticated malware cases
  • Understand real advanced attacks, covering all stages from infiltration to hacking the system
  • Learn to bypass anti-reverse engineering techniques

Who this book is for

If you are an IT security administrator, forensic analyst, or malware researcher looking to secure against malicious software or investigate malicious code, this book is for you. Prior programming experience and a fair understanding of malware attacks and investigation is expected.

Table of contents

  1. Title Page
  2. Copyright and Credits
    1. Mastering Malware Analysis
  3. About Packt
    1. Why subscribe?
  4. Contributors
    1. About the authors
    2. About the reviewers
    3. Packt is searching for authors like you
  5. Preface
    1. Who this book is for
    2. What this book covers
    3. To get the most out of this book
    4. Download the example code files
    5. Download the color images
    6. Conventions used
    7. Get in touch
    8. Reviews
  6. Section 1: Fundamental Theory
  7. A Crash Course in CISC/RISC and Programming Basics
    1. Basic concepts
    2. Registers
    3. Memory
    4. Virtual memory
    5. Stack
    6. Branches, loops, and conditions
    7. Exceptions, interrupts, and communicating with other devices
    8. Assembly languages
    9. CISC versus RISC
    10. Types of instructions
    11. Becoming familiar with x86 (IA-32 and x64)
    12. Registers
    13. Special registers
    14. The instruction structure
    15. opcode
    16. dest
    17. src
    18. The instruction set
    19. Data manipulation instructions
    20. Data transfer instructions
    21. Flow control instructions
    22. Arguments, local variables, and calling conventions (in x86 and x64)
    23. stdcall
    24. Arguments
    25. Local variables
    26. cdecl
    27. fastcall
    28. thiscall
    29. The x64 calling convention
    30. Exploring ARM assembly
    31. Basics
    32. Instruction sets
    33. Basics of MIPS
    34. Basics
    35. The instruction set
    36. Diving deep into PowerPC
    37. Basics
    38. The instruction set
    39. Covering the SuperH assembly
    40. Basics
    41. The instruction set
    42. Working with SPARC
    43. Basics
    44. The instruction set
    45. Moving from assembly to high-level programming languages
    46. Arithmetic statements
    47. If conditions
    48. While loop conditions
    49. Summary
  8. Section 2: Diving Deep into Windows Malware
  9. Basic Static and Dynamic Analysis for x86/x64
    1. Working with the PE header structure
    2. Why PE?
    3. Exploring PE structure
    4. MZ header
    5. PE header
    6. File header
    7. Optional header
    8. Data directory
    9. Section table
    10. PE+ (x64 PE)
    11. PE header analysis tools
    12. Static and dynamic linking
    13. Static linking
    14. Dynamic linking
    15. Dynamic link libraries
    16. Application programming interface
    17. Dynamic API loading
    18. Using PE header information for static analysis
    19. How to use PE header for incident handling
    20. How to use a PE header for threat intelligence
    21. PE loading and process creation
    22. Basic terminology
    23. What's process?
    24. Virtual memory to physical memory mapping
    25. Threads
    26. Important data structures: TIB, TEB, and PEB
    27. Process loading step by step
    28. PE file loading step by step
    29. WOW64 processes
    30. Dynamic analysis with OllyDbg/Immunity Debugger
    31. Debugging tools
    32. How to analyze a sample with OllyDbg
    33. Types of breakpoints
    34. Step into/step over breakpoint
    35. INT3 breakpoint
    36. Memory breakpoints
    37. Hardware breakpoints
    38. Modifying the program execution
    39. Patching—modifying the program's assembly instructions
    40. Change EFlags
    41. Modifying the instruction pointer value
    42. Changing the program data
    43. Debugging malicious services
    44. What is service?
    45. Attaching to the service
    46. Summary
  10. Unpacking, Decryption, and Deobfuscation
    1. Exploring packers
    2. Exploring packing and encrypting tools
    3. Identifying a packed sample
    4. Technique 1 – checking PE tool static signatures
    5. Technique 2 – evaluating PE section names
    6. Technique 3 – using stub execution signs
    7. Technique 4 – detecting a small import table
    8. Automatically unpacking packed samples
    9. Technique 1 – the official unpacking process
    10. Technique 2 – using OllyScript with OllyDbg
    11. Technique 3 – using generic unpackers
    12. Technique 4 – emulation
    13. Technique 5 – memory dumps
    14. Manual unpacking using OllyDbg
    15. Technique 6 – memory breakpoint on execution
    16. Step 1 – setting the breakpoints
    17. Step 2 – turning on Data Execution Prevention
    18. Step 3 – preventing any further attempts to change memory permissions
    19. Step 4 – executing and getting the OEP
    20. Technique 7 – call stack backtracing
    21. Step 1 – setting the breakpoints
    22. Step 2 – following the call stack
    23. Step 3 – reaching the OEP
    24. Technique 8 – monitoring memory allocated spaces for unpacked code
    25. Technique 9 – in-place unpacking
    26. Technique 10 – stack restoration-based
    27. Dumping the unpacked sample and fixing the import table
    28. Dumping the process
    29. Fixing the import table
    30. Identifying different encryption algorithms and functions
    31. Types of encryption algorithms
    32. Basic encryption algorithms
    33. How to identify encryption functions
    34. String search detection techniques for simple algorithms
    35. The basics of X-RAYING
    36. Simple static encryption
    37. Other encryption algorithms
    38. X-RAYING tools for malware analysis and detection
    39. Identifying the RC4 encryption algorithm
    40. The RC4 encryption algorithm
    41. Key-scheduling algorithm
    42. Pseudo-random generation algorithm
    43. Identifying RC4 algorithms in a malware sample
    44. Standard symmetric and asymmetric encryption algorithms
    45. Extracting information from Windows cryptography APIs
    46. Step 1 – initializing and connecting to the cryptographic service provider (CSP)
    47. Step 2 – preparing the key
    48. Step 3 – encrypting or decrypting the data
    49. Step 4 – freeing the memory
    50. Cryptography API next generation (CNG)
    51. Applications of encryption in modern malware – Vawtrak banking Trojan
    52. String and API name encryption
    53. Network communication encryption
    54. Using IDA for decryption and unpacking
    55. IDA tips and tricks
    56. Static analysis
    57. Dynamic analysis
    58. Classic and new syntax of IDA scripts
    59. Dynamic string decryption
    60. Dynamic WinAPIs resolution
    61. Summary
  11. Inspecting Process Injection and API Hooking
    1. Understanding process injection
    2. What's process injection?
    3. Why process injection?
    4. DLL injection
    5. Windows-supported DLL injection
    6. A simple DLL injection technique
    7. Working with process injection
    8. Getting the list of running processes
    9. Code injection
    10. Advanced code injection-reflective DLL injection
    11. Stuxnet secret technique-process hollowing
    12. Dynamic analysis of code injection
    13. Technique 1—debug it where it is
    14. Technique 2—attach to the targeted process
    15. Technique 3—dealing with process hollowing
    16. Memory forensics techniques for process injection
    17. Technique 1—detecting code injection and reflective DLL injection 
    18. Technique 2—detecting process hollowing
    19. Technique 3—detecting process hollowing using the HollowFind plugin
    20. Understanding API hooking
    21. Why API hooking?
    22. Working with API hooking
    23. Inline API hooking
    24. Inline API hooking with trampoline
    25. Inline API hooking with a length disassembler
    26. Detecting API hooking using memory forensics
    27. Exploring IAT hooking
    28. Summary
  12. Bypassing Anti-Reverse Engineering Techniques
    1. Exploring debugger detection
    2. Direct check for debugger presence
    3. Detecting a debugger through an environment change
    4. Detecting a debugger using parent processes
    5. Handling debugger breakpoints evasion
    6. Detecting software breakpoints (INT3)
    7. Detecting single-stepping breakpoints (trap flag)
    8. Detecting a trap flag using the SS register
    9. Detecting single-stepping using timing techniques
    10. Evading hardware breakpoints
    11. What is structured exception handling?
    12. Detecting and removing hardware breakpoints
    13. Memory breakpoints
    14. Escaping the debugger
    15. Process injection
    16. TLS callbacks
    17. Windows events callbacks
    18. Obfuscation and anti-disassemblers
    19. Encryption
    20. Junk code insertion
    21. Code transportation
    22. Dynamic API calling with checksum
    23. Proxy functions and proxy argument stacking
    24. Detecting and evading behavioral analysis tools
    25. Finding the tool process
    26. Searching for the tool window
    27. Detecting sandboxes and virtual machines
    28. Different output between virtual machines and real machines
    29. Detecting virtualization processes and services
    30. Detecting virtualization through registry keys
    31. Detecting virtual machines using PowerShell
    32. Detecting sandboxes by using default settings
    33. Other techniques
    34. Summary
  13. Understanding Kernel-Mode Rootkits
    1. Kernel mode versus user mode
    2. Protection rings
    3. Windows internals
    4. The infrastructure of Windows
    5. The execution path from user mode to kernel mode
    6. Rootkits and device drivers
    7. What is a rootkit?
    8. Types of rootkits
    9. What is a device driver?
    10. Hooking mechanisms
    11. SSDT hooking
    12. Hooking the SYSENTER entry function
    13. Modifying SSDT in an x86 environment
    14. Modifying SSDT in an x64 environment
    15. Hooking SSDT functions
    16. IRP hooking
    17. Devices and major functions
    18. Attaching to a device
    19. Modifying the IRP response and setting a completion routine
    20. DKOM
    21. The kernel objects—EPROCESS and ETHREAD
    22. How do rootkits perform an object manipulation attack?
    23. Process injection in kernel mode
    24. Executing the inject code using APC queuing
    25. KPP in x64 systems (PatchGuard)
    26. Bypassing driver signature enforcement
    27. Bypassing PatchGuard—the Turla example
    28. Bypassing PatchGuard—GhostHook
    29. Disabling PatchGuard using the Command Prompt
    30. Static and dynamic analysis in kernel mode
    31. Static analysis
    32. Tools
    33. Tips and tricks
    34. Dynamic and behavioral analysis
    35. Tools
    36. Monitors
    37. Rootkit detectors
    38. Setting up a testing environment
    39. Setting up the debugger
    40. Stopping at the driver's entry point
    41. Loading the driver
    42. Restoring the debugging state
    43. Summary
  14. Section 3: Examining Cross-Platform Malware
  15. Handling Exploits and Shellcode
    1. Getting familiar with vulnerabilities and exploits
    2. Types of vulnerabilities
    3. Stack overflow vulnerability
    4. Heap overflow vulnerabilities
    5. The use-after-free vulnerability
    6. Logical vulnerabilities
    7. Types of exploits
    8. Cracking the shellcode
    9. What's shellcode?
    10. Linux shellcode in x86-64
    11. Getting the absolute address
    12. Null-free shellcode
    13. Local shell shellcode
    14. Reverse shell shellcode
    15. Linux shellcode for ARM
    16. Null-free shellcode
    17. Windows shellcode
    18. Getting the Kernel32.dll's ImageBase
    19. Getting the required APIs from Kernel32.dll
    20. The download and execute shellcode
    21. Static and dynamic analysis of exploits
    22. Analysis workflow
    23. Shellcode analysis
    24. Exploring bypasses for exploit mitigation technologies
    25. Data execution prevention (DEP/NX)
    26. Return-oriented programming
    27. Address space layout randomization
    28. DEP and partial ASLR
    29. DEP and full ASLR – partial ROP and chaining multiple vulnerabilities
    30. DEP and full ASLR – heap spray technique
    31. Other mitigation technologies
    32. Analyzing Microsoft Office exploits
    33. File structures
    34. Compound file binary format
    35. Rich text format
    36. Office open XML format
    37. Static and dynamic analysis of MS Office exploits
    38. Static analysis
    39. Dynamic analysis
    40. Studying malicious PDFs
    41. File structure
    42. Static and dynamic analysis of PDF files
    43. Static analysis
    44. Dynamic analysis
    45. Summary
  16. Reversing Bytecode Languages: .NET, Java, and More
    1. The basic theory of bytecode languages
    2. Object-oriented programming
    3. Inheritance
    4. Polymorphism
    5. .NET explained
    6. .NET file structure
    7. .NET COR20 header
    8. Metadata streams
    9. How to identify a .NET application from PE characteristics
    10. The CIL language instruction set
    11. Pushing into stack instructions
    12. Pulling out a value from the stack
    13. Mathematical and logical operations
    14. Branching instructions
    15. CIL language to higher-level languages
    16. Local variable assignments
    17. Local variable assignment with a method return value
    18. Basic branching statements
    19. Loops statements
    20. .NET malware analysis
    21. .NET analysis tools
    22. Static and dynamic analysis (with Dnspy)
    23. .NET static analysis
    24. .NET dynamic analysis
    25. Patching a .NET sample
    26. Dealing with obfuscation
    27. Obfuscated names for classes, methods, and others
    28. Encrypted strings inside the binary
    29. The sample is obfuscated using an obfuscator
    30. The essentials of Visual Basic
    31. File structure
    32. P-code versus native code
    33. Common p-code instructions
    34. Dissecting Visual Basic samples
    35. Static analysis
    36. P-code
    37. Native code
    38. Dynamic analysis
    39. P-code
    40. Native code
    41. The internals of Java samples
    42. File structure
    43. JVM instructions
    44. Static analysis
    45. Dynamic analysis
    46. Dealing with anti-reverse engineering solutions
    47. Python—script language internals
    48. File structure
    49. Bytecode instructions
    50. Analyzing compiled Python
    51. Static analysis
    52. Dynamic analysis
    53. Summary
  17. Scripts and Macros: Reversing, Deobfuscation, and Debugging
    1. Classic shell script languages
    2. Windows batch scripting
    3. Bash
    4. VBScript explained
    5. Basic syntax
    6. Static and dynamic analysis
    7. Deobfuscation
    8. Those evil macros inside documents
    9. Basic syntax
    10. Static and dynamic analysis
    11. Besides macros
    12. The power of PowerShell
    13. Basic syntax
    14. Static and dynamic analysis
    15. Handling JavaScript
    16. Basic syntax
    17. Static and dynamic analysis
    18. Anti-reverse engineering tricks
    19. Behind C and C—even malware has its own backend
    20. Things to focus on
    21. Static and dynamic analysis
    22. Other script languages
    23. Where to start from
    24. Questions to answer
    25. Summary
  18. Section 4: Looking into IoT and Other Platforms
  19. Dissecting Linux and IoT Malware
    1. Explaining ELF files 
    2. ELF structure
    3. System calls
    4. Filesystem
    5. Network
    6. Process management
    7. Other
    8. Syscalls in assembly
    9. Common anti-reverse engineering tricks
    10. Exploring common behavioral patterns
    11. Initial delivery and lateral movement
    12. Persistence
    13. Privilege escalation
    14. Interaction with the command and control server
    15. Attacking stage
    16. Static and dynamic analysis of x86 (32- and 64-bit) samples
    17. Static analysis
    18. File type detectors
    19. Data carving
    20. Disassemblers
    21. Actual tools
    22. Engines
    23. How to choose
    24. Dynamic analysis
    25. Tracers
    26. Network monitors
    27. Debuggers
    28. Binary emulators
    29. Radare2 cheat sheet
    30. Anti-reverse engineering techniques
    31. Learning Mirai, its clones, and more
    32. High-level functionality
    33. Propagation
    34. Weaponry
    35. Self-defense
    36. Later derivatives
    37. Other widespread families
    38. Static and dynamic analysis of RISC samples
    39. ARM
    40. MIPS
    41. PowerPC
    42. SuperH
    43. SPARC
    44. Handling other architectures
    45. What to start from
    46. Summary
  20. Introduction to macOS and iOS Threats
    1. Understanding the role of the security model
    2. macOS
    3. Security policies
    4. Filesystem hierarchy and encryption
    5. Directory structure
    6. Encryption
    7. Apps protection
    8. Gatekeeper
    9. App sandbox
    10. Other technologies
    11. iOS
    12. System security
    13. Data encryption and password management
    14. Apps' security
    15. File formats and APIs
    16. Mach-O
    17. Thin
    18. Fat
    19. Application bundles (.app)
    20. Info.plist
    21. macOS
    22. iOS
    23. Installer packages (.pkg)
    24. Apple disk images (.dmg)
    25. iOS app store packages (.ipa)
    26. APIs
    27. Static and dynamic analyses of macOS and iOS samples
    28. Static analysis
    29. Retrieving samples
    30. Disassemblers and decompilers
    31. Auxiliary tools and libraries
    32. Dynamic and behavioral analysis
    33. macOS
    34. Debuggers
    35. Monitoring and dynamic instrumentation
    36. Network analysis
    37. iOS
    38. Installers and loaders
    39. Debuggers
    40. Dumping and decryption
    41. Monitors and in-memory patching
    42. Network analysis
    43. Attack stages
    44. Jailbreaks on demand
    45. Penetration
    46. Deployment and persistence
    47. macOS
    48. iOS
    49. Action phase
    50. macOS
    51. iOS
    52. Other attack techniques
    53. macOS
    54. iOS
    55. Advanced techniques
    56. Anti-reverse-engineering (RE) tricks
    57. Misusing dynamic data exchange (DDE)
    58. User hiding
    59. Use of AppleScript
    60. API hijacking
    61. Rootkits for Mac—do they exist?
    62. Analysis workflow
    63. Summary
  21. Analyzing Android Malware Samples
    1. (Ab)using Android internals 
    2. File hierarchy
    3. Android security model
    4. Process management
    5. Filesystem
    6. App permissions
    7. Security services
    8. Console
    9. To root or not to root?
    10. Understanding Dalvik and ART 
    11. Dalvik VM (DVM)
    12. Android runtime (ART)
    13. APIs
    14. File formats
    15. DEX
    16. ODEX
    17. OAT
    18. VDEX
    19. ART
    20. ELF
    21. APK
    22. Bytecode set
    23. Malware behavior patterns
    24. Attack stages
    25. Penetration
    26. Deployment
    27. Action phase
    28. Advanced techniques—investment pays off
    29. Patching system libraries
    30. Keylogging
    31. Self-defense
    32. Rootkits—get it covered
    33. Static and dynamic analysis of threats
    34. Static analysis
    35. Disassembling and data extraction
    36. Decompiling
    37. Dynamic analysis
    38. Android debug bridge
    39. Emulators
    40. Behavioral analysis and tracing
    41. Debuggers
    42. Analysis workflow
    43. Summary
  22. Other Books You May Enjoy
    1. Leave a review - let other readers know what you think

Product information

  • Title: Mastering Malware Analysis
  • Author(s): Alexey Kleymenov, Amr Thabet
  • Release date: June 2019
  • Publisher(s): Packt Publishing
  • ISBN: 9781789610789