Learning Linux Binary Analysis

Learning Linux Binary Analysis

by Ryan elfmaster O'Neill
Released February 2016
Publisher(s): Packt Publishing
ISBN: 9781782167105

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial

Book description

Uncover the secrets of Linux binary analysis with this handy guide

About This Book

  • Grasp the intricacies of the ELF binary format of UNIX and Linux
  • Design tools for reverse engineering and binary forensic analysis
  • Insights into UNIX and Linux memory infections, ELF viruses, and binary protection schemes

Who This Book Is For

If you are a software engineer or reverse engineer and want to learn more about Linux binary analysis, this book will provide you with all you need to implement solutions for binary analysis in areas of security, forensics, and antivirus. This book is great for both security enthusiasts and system level engineers. Some experience with the C programming language and the Linux command line is assumed.

What You Will Learn

  • Explore the internal workings of the ELF binary format
  • Discover techniques for UNIX Virus infection and analysis
  • Work with binary hardening and software anti-tamper methods
  • Patch executables and process memory
  • Bypass anti-debugging measures used in malware
  • Perform advanced forensic analysis of binaries
  • Design ELF-related tools in the C language
  • Learn to operate on memory with ptrace

In Detail

Learning Linux Binary Analysis is packed with knowledge and code that will teach you the inner workings of the ELF format, and the methods used by hackers and security analysts for virus analysis, binary patching, software protection and more.

This book will start by taking you through UNIX/Linux object utilities, and will move on to teaching you all about the ELF specimen. You will learn about process tracing, and will explore the different types of Linux and UNIX viruses, and how you can make use of ELF Virus Technology to deal with them.

The latter half of the book discusses the usage of Kprobe instrumentation for kernel hacking, code patching, and debugging. You will discover how to detect and disinfect kernel-mode rootkits, and move on to analyze static code. Finally, you will be walked through complex userspace memory infection analysis.

This book will lead you into territory that is uncharted even by some experts; right into the world of the computer hacker.

Style and approach

The material in this book provides detailed insight into the arcane arts of hacking, coding, reverse engineering Linux executables, and dissecting process memory. In the computer security industry these skills are priceless, and scarce. The tutorials are filled with knowledge gained through first hand experience, and are complemented with frequent examples including source code.

Table of contents

  1. Learning Linux Binary Analysis
    1. Table of Contents
    2. Learning Linux Binary Analysis
    3. Credits
    4. About the Author
    5. Acknowledgments
    6. About the Reviewers
    7. www.PacktPub.com
      1. Support files, eBooks, discount offers, and more
        1. Why subscribe?
        2. Free access for Packt account holders
    8. Preface
      1. What this book covers
      2. What you need for this book
      3. Who this book is for
      4. Conventions
      5. Reader feedback
      6. Customer support
        1. Downloading the example code
        2. Errata
        3. Piracy
        4. Questions
    9. 1. The Linux Environment and Its Tools
      1. Linux tools
        1. GDB
        2. Objdump from GNU binutils
        3. Objcopy from GNU binutils
        4. strace
        5. ltrace
        6. Basic ltrace command
        7. ftrace
        8. readelf
        9. ERESI – The ELF reverse engineering system interface
      2. Useful devices and files
        1. /proc/<pid>/maps
        2. /proc/kcore
        3. /boot/System.map
        4. /proc/kallsyms
        5. /proc/iomem
        6. ECFS
      3. Linker-related environment points
        1. The LD_PRELOAD environment variable
        2. The LD_SHOW_AUXV environment variable
        3. Linker scripts
      4. Summary
    10. 2. The ELF Binary Format
      1. ELF file types
      2. ELF program headers
        1. PT_LOAD
        2. PT_DYNAMIC – Phdr for the dynamic segment
        3. PT_NOTE
        4. PT_INTERP
        5. PT_PHDR
      3. ELF section headers
        1. The .text section
        2. The .rodata section
        3. The .plt section
        4. The .data section
        5. The .bss section
        6. The .got.plt section
        7. The .dynsym section
        8. The .dynstr section
        9. The .rel.* section
        10. The .hash section
        11. The .symtab section
        12. The .strtab section
        13. The .shstrtab section
        14. The .ctors and .dtors sections
      4. ELF symbols
        1. st_name
        2. st_value
        3. st_size
        4. st_other
        5. st_shndx
        6. st_info
          1. Symbol types
          2. Symbol bindings
      5. ELF relocations
        1. Relocatable code injection-based binary patching
      6. ELF dynamic linking
        1. The auxiliary vector
        2. Learning about the PLT/GOT
        3. The dynamic segment revisited
          1. DT_NEEDED
          2. DT_SYMTAB
          3. DT_HASH
          4. DT_STRTAB
          5. DT_PLTGOT
      7. Coding an ELF Parser
      8. Summary
    11. 3. Linux Process Tracing
      1. The importance of ptrace
      2. ptrace requests
        1. ptrace request types
      3. The process register state and flags
      4. A simple ptrace-based debugger
        1. Using the tracer program
      5. A simple ptrace debugger with process attach capabilities
      6. Advanced function-tracing software
      7. ptrace and forensic analysis
        1. What to look for in the memory
      8. Process image reconstruction – from the memory to the executable
        1. Challenges for process-executable reconstruction
        2. Challenges for executable reconstruction
          1. PLT/GOT integrity
        3. Adding a section header table
        4. The algorithm for the process
        5. Process reconstruction with Quenya on a 32-bit test environment
      9. Code injection with ptrace
      10. Simple examples aren't always so trivial
      11. Demonstrating the code_inject tool
      12. A ptrace anti-debugging trick
        1. Is your program being traced?
      13. Summary
    12. 4. ELF Virus Technology – Linux/Unix Viruses
      1. ELF virus technology
      2. ELF virus engineering challenges
        1. Parasite code must be self-contained
          1. Solution
        2. Complications with string storage
          1. Solution
        3. Finding legitimate space to store parasite code
          1. Solution
        4. Passing the execution control flow to the parasite
          1. Solution
      3. ELF virus parasite infection methods
        1. The Silvio padding infection method
          1. Algorithm for the Silvio .text infection method
          2. An example of text segment padding infection
            1. Adjusting the ELF headers
            2. Inserting the parasite code
          3. Example of using the functions above
          4. The LPV virus
          5. Use cases for the Silvio padding infection
        2. The reverse text infection
          1. Algorithm for reverse text infection
        3. Data segment infections
          1. Algorithm for data segment infection
      4. The PT_NOTE to PT_LOAD conversion infection method
        1. Algorithm for PT_NOTE to PT_LOAD conversion infections
      5. Infecting control flow
        1. Direct PLT infection
        2. Function trampolines
        3. Overwriting the .ctors/.dtors function pointers
        4. GOT – global offset table poisoning or PLT/GOT redirection
        5. Infecting data structures
        6. Function pointer overwrites
      6. Process memory viruses and rootkits – remote code injection techniques
        1. Shared library injection – .so injection/ET_DYN injection
        2. .so injection with LD_PRELOAD
          1. Illustration 4.7 – using LD_PRELOAD to inject wicked.so.1
        3. .so injection with open()/mmap() shellcode
        4. .so injection with dlopen() shellcode
          1. Illustration 4.8 – C code invoking __libc_dlopen_mode()
        5. .so injection with VDSO manipulation
        6. Text segment code injections
        7. Executable injections
        8. Relocatable code injection – the ET_REL injection
      7. ELF anti-debugging and packing techniques
        1. The PTRACE_TRACEME technique
          1. Illustration 4.9 – an anti-debug with PTRACE_TRACEME example
        2. The SIGTRAP handler technique
        3. The /proc/self/status technique
        4. The code obfuscation technique
        5. The string table transformation technique
      8. ELF virus detection and disinfection
      9. Summary
    13. 5. Linux Binary Protection
      1. ELF binary packers – dumb protectors
      2. Stub mechanics and the userland exec
        1. An example of a protector
      3. Other jobs performed by protector stubs
      4. Existing ELF binary protectors
        1. DacryFile by the Grugq – 2001
        2. Burneye by Scut – 2002
        3. Shiva by Neil Mehta and Shawn Clowes – 2003
        4. Maya's Veil by Ryan O'Neill – 2014
          1. Maya's protection layers
            1. Layer 1
            2. Layer 2
            3. Layer 3
          2. Maya's nanomites
          3. Maya's anti-exploitation
            1. Source code of vuln.c
            2. Example of exploiting vuln.c
      5. Downloading Maya-protected binaries
      6. Anti-debugging for binary protection
      7. Resistance to emulation
        1. Detecting emulation through syscall testing
        2. Detecting emulated CPU inconsistencies
        3. Checking timing delays between certain instructions
      8. Obfuscation methods
      9. Protecting control flow integrity
        1. Attacks based on ptrace
        2. Security vulnerability-based attacks
      10. Other resources
      11. Summary
    14. 6. ELF Binary Forensics in Linux
      1. The science of detecting entry point modification
      2. Detecting other forms of control flow hijacking
        1. Patching the .ctors/.init_array section
        2. Detecting PLT/GOT hooks
          1. Truncated output from readelf -S command
        3. Detecting function trampolines
      3. Identifying parasite code characteristics
      4. Checking the dynamic segment for DLL injection traces
      5. Identifying reverse text padding infections
      6. Identifying text segment padding infections
      7. Identifying protected binaries
        1. Analyzing a protected binary
      8. IDA Pro
      9. Summary
    15. 7. Process Memory Forensics
      1. What does a process look like?
        1. Executable memory mappings
        2. The program heap
        3. Shared library mappings
        4. The stack, vdso, and vsyscall
      2. Process memory infection
        1. Process infection tools
        2. Process infection techniques
          1. Injection methods
          2. Techniques for hijacking execution
      3. Detecting the ET_DYN injection
        1. Azazel userland rootkit detection
        2. Mapping out the process address space
        3. Finding LD_PRELOAD on the stack
        4. Detecting PLT/GOT hooks
          1. Identifying incorrect GOT addresses
        5. ET_DYN injection internals
          1. Example – finding the symbol for __libc_dlopen_mode
          2. Code example – the __libc_dlopen_mode shellcode
          3. Code example – libc symbol resolution
          4. Code example – the x86_32 shellcode to mmap() an ET_DYN object
        6. Manipulating VDSO to perform dirty work
        7. Shared object loading – legitimate or not?
          1. Legitimate shared object loading
          2. Illegitimate shared object loading
        8. Heuristics for .so injection detection
        9. Tools for detecting PLT/GOT hooks
      4. Linux ELF core files
        1. Analysis of the core file – the Azazel rootkit
          1. Starting up an Azazel infected process and getting a core dump
          2. Core file program headers
          3. The PT_NOTE segment
          4. PT_LOAD segments and the downfalls of core files for forensics purposes
          5. Using a core file with GDB for forensics
      5. Summary
    16. 8. ECFS – Extended Core File Snapshot Technology
      1. History
      2. The ECFS philosophy
      3. Getting started with ECFS
        1. Plugging ECFS into the core handler
        2. ECFS snapshots without killing the process
      4. libecfs – a library for parsing ECFS files
      5. readecfs
      6. Examining an infected process using ECFS
        1. Infecting the host process
        2. Capturing and analyzing an ECFS snapshot
          1. The symbol table analysis
          2. The section header analysis
        3. Extracting parasite code with readecfs
        4. Analyzing the Azazel userland rootkit
          1. The symbol table of the host2 process reconstructed
          2. The section header table of the host2 process reconstructed
          3. Validating the PLT/GOT with ECFS
          4. The readecfs output for PLT/GOT validation
      7. The ECFS reference guide
        1. ECFS symbol table reconstruction
        2. ECFS section headers
        3. Using an ECFS file as a regular core file
        4. The libecfs API and how to use it
      8. Process necromancy with ECFS
      9. Learning more about ECFS
      10. Summary
    17. 9. Linux /proc/kcore Analysis
      1. Linux kernel forensics and rootkits
      2. stock vmlinux has no symbols
        1. Building a proper vmlinux with kdress
      3. /proc/kcore and GDB exploration
        1. An example of navigating sys_call_table
      4. Direct sys_call_table modifications
        1. Detecting sys_call_table modifications
          1. An example of validating the integrity of a syscall
        2. Kernel function trampolines
        3. Example of function trampolines
          1. An example code for hijacking sys_write on a 32-bit kernel
        4. Detecting function trampolines
          1. An example with the ret instruction
          2. An example with indirect jmp
          3. An example with relative jmp
          4. Interrupt handler patching – int 0x80, syscall
        5. Detecting interrupt handler patching
      5. Kprobe rootkits
        1. Detecting kprobe rootkits
      6. Debug register rootkits – DRR
        1. Detecting DRR
      7. VFS layer rootkits
        1. Detecting VFS layer rootkits
          1. An example of validating a VFS function pointer
      8. Other kernel infection techniques
      9. vmlinux and .altinstructions patching
        1. .altinstructions and .altinstr_replace
        2. From arch/x86/include/asm/alternative.h
        3. Using textify to verify kernel code integrity
        4. An example of using textify to check sys_call_table
      10. Using taskverse to see hidden processes
        1. Taskverse techniques
      11. Infected LKMs – kernel drivers
        1. Method 1 for infecting LKM files – symbol hijacking
        2. Method 2 for infecting LKM files (function hijacking)
        3. Detecting infected LKMs
      12. Notes on /dev/kmem and /dev/mem
      13. /dev/mem
        1. FreeBSD /dev/kmem
      14. K-ecfs – kernel ECFS
        1. A sneak peek of the kernel-ecfs file
      15. Kernel hacking goodies
        1. General reverse engineering and debugging
        2. Advanced kernel hacking/debugging interfaces
        3. Papers mentioned in this chapter
      16. Summary
    18. Index

Product information

  • Title: Learning Linux Binary Analysis
  • Author(s): Ryan elfmaster O'Neill
  • Release date: February 2016
  • Publisher(s): Packt Publishing
  • ISBN: 9781782167105