book

Mastering Malware Analysis

Name: Mastering Malware Analysis
ISBN: 9781789610789

by Alexey Kleymenov, Amr Thabet

June 2019

Beginner

562 pages

11h 36m

English

Packt Publishing

Read now

Unlock full access

Title Page
Copyright and Credits
Mastering Malware Analysis
About Packt
Why subscribe?
Contributors
About the authorsAbout the reviewersPackt is searching for authors like you
Preface
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchReviews
Section 1: Fundamental Theory
A Crash Course in CISC/RISC and Programming Basics
Basic conceptsRegistersMemoryVirtual memoryStackBranches, loops, and conditionsExceptions, interrupts, and communicating with other devicesAssembly languagesCISC versus RISCTypes of instructionsBecoming familiar with x86 (IA-32 and x64)RegistersSpecial registersThe instruction structureopcodedestsrcThe instruction setData manipulation instructionsData transfer instructionsFlow control instructionsArguments, local variables, and calling conventions (in x86 and x64)stdcallArgumentsLocal variablescdeclfastcallthiscallThe x64 calling conventionExploring ARM assemblyBasicsInstruction setsBasics of MIPSBasicsThe instruction setDiving deep into PowerPCBasicsThe instruction setCovering the SuperH assemblyBasicsThe instruction setWorking with SPARCBasicsThe instruction setMoving from assembly to high-level programming languagesArithmetic statementsIf conditionsWhile loop conditionsSummary
Section 2: Diving Deep into Windows Malware
Basic Static and Dynamic Analysis for x86/x64
Working with the PE header structureWhy PE?Exploring PE structureMZ headerPE headerFile headerOptional headerData directorySection tablePE+ (x64 PE)PE header analysis toolsStatic and dynamic linkingStatic linkingDynamic linkingDynamic link librariesApplication programming interfaceDynamic API loadingUsing PE header information for static analysisHow to use PE header for incident handlingHow to use a PE header for threat intelligencePE loading and process creationBasic terminologyWhat's process?Virtual memory to physical memory mappingThreadsImportant data structures: TIB, TEB, and PEBProcess loading step by stepPE file loading step by stepWOW64 processesDynamic analysis with OllyDbg/Immunity DebuggerDebugging toolsHow to analyze a sample with OllyDbgTypes of breakpointsStep into/step over breakpointINT3 breakpointMemory breakpointsHardware breakpointsModifying the program executionPatching—modifying the program's assembly instructionsChange EFlagsModifying the instruction pointer valueChanging the program dataDebugging malicious servicesWhat is service?Attaching to the serviceSummary
Unpacking, Decryption, and Deobfuscation
Exploring packersExploring packing and encrypting toolsIdentifying a packed sampleTechnique 1 – checking PE tool static signaturesTechnique 2 – evaluating PE section namesTechnique 3 – using stub execution signsTechnique 4 – detecting a small import tableAutomatically unpacking packed samplesTechnique 1 – the official unpacking processTechnique 2 – using OllyScript with OllyDbgTechnique 3 – using generic unpackersTechnique 4 – emulationTechnique 5 – memory dumpsManual unpacking using OllyDbgTechnique 6 – memory breakpoint on executionStep 1 – setting the breakpointsStep 2 – turning on Data Execution PreventionStep 3 – preventing any further attempts to change memory permissionsStep 4 – executing and getting the OEPTechnique 7 – call stack backtracingStep 1 – setting the breakpointsStep 2 – following the call stackStep 3 – reaching the OEPTechnique 8 – monitoring memory allocated spaces for unpacked codeTechnique 9 – in-place unpackingTechnique 10 – stack restoration-basedDumping the unpacked sample and fixing the import tableDumping the processFixing the import tableIdentifying different encryption algorithms and functionsTypes of encryption algorithmsBasic encryption algorithmsHow to identify encryption functionsString search detection techniques for simple algorithmsThe basics of X-RAYINGSimple static encryptionOther encryption algorithmsX-RAYING tools for malware analysis and detectionIdentifying the RC4 encryption algorithmThe RC4 encryption algorithmKey-scheduling algorithmPseudo-random generation algorithmIdentifying RC4 algorithms in a malware sampleStandard symmetric and asymmetric encryption algorithmsExtracting information from Windows cryptography APIsStep 1 – initializing and connecting to the cryptographic service provider (CSP)Step 2 – preparing the keyStep 3 – encrypting or decrypting the dataStep 4 – freeing the memoryCryptography API next generation (CNG)Applications of encryption in modern malware – Vawtrak banking TrojanString and API name encryptionNetwork communication encryptionUsing IDA for decryption and unpackingIDA tips and tricksStatic analysisDynamic analysisClassic and new syntax of IDA scriptsDynamic string decryptionDynamic WinAPIs resolutionSummary

Inspecting Process Injection and API Hooking
Understanding process injectionWhat's process injection?Why process injection?DLL injectionWindows-supported DLL injectionA simple DLL injection techniqueWorking with process injectionGetting the list of running processesCode injectionAdvanced code injection-reflective DLL injectionStuxnet secret technique-process hollowingDynamic analysis of code injectionTechnique 1—debug it where it isTechnique 2—attach to the targeted processTechnique 3—dealing with process hollowingMemory forensics techniques for process injectionTechnique 1—detecting code injection and reflective DLL injection Technique 2—detecting process hollowingTechnique 3—detecting process hollowing using the HollowFind pluginUnderstanding API hookingWhy API hooking?Working with API hookingInline API hookingInline API hooking with trampolineInline API hooking with a length disassemblerDetecting API hooking using memory forensicsExploring IAT hookingSummary
Bypassing Anti-Reverse Engineering Techniques
Exploring debugger detectionDirect check for debugger presenceDetecting a debugger through an environment changeDetecting a debugger using parent processesHandling debugger breakpoints evasionDetecting software breakpoints (INT3)Detecting single-stepping breakpoints (trap flag)Detecting a trap flag using the SS registerDetecting single-stepping using timing techniquesEvading hardware breakpointsWhat is structured exception handling?Detecting and removing hardware breakpointsMemory breakpointsEscaping the debuggerProcess injectionTLS callbacksWindows events callbacksObfuscation and anti-disassemblersEncryptionJunk code insertionCode transportationDynamic API calling with checksumProxy functions and proxy argument stackingDetecting and evading behavioral analysis toolsFinding the tool processSearching for the tool windowDetecting sandboxes and virtual machinesDifferent output between virtual machines and real machinesDetecting virtualization processes and servicesDetecting virtualization through registry keysDetecting virtual machines using PowerShellDetecting sandboxes by using default settingsOther techniquesSummary
Understanding Kernel-Mode Rootkits
Kernel mode versus user modeProtection ringsWindows internalsThe infrastructure of WindowsThe execution path from user mode to kernel modeRootkits and device driversWhat is a rootkit?Types of rootkitsWhat is a device driver?Hooking mechanismsSSDT hookingHooking the SYSENTER entry functionModifying SSDT in an x86 environmentModifying SSDT in an x64 environmentHooking SSDT functionsIRP hookingDevices and major functionsAttaching to a deviceModifying the IRP response and setting a completion routineDKOMThe kernel objects—EPROCESS and ETHREADHow do rootkits perform an object manipulation attack?Process injection in kernel modeExecuting the inject code using APC queuingKPP in x64 systems (PatchGuard)Bypassing driver signature enforcementBypassing PatchGuard—the Turla exampleBypassing PatchGuard—GhostHookDisabling PatchGuard using the Command PromptStatic and dynamic analysis in kernel modeStatic analysisToolsTips and tricksDynamic and behavioral analysisToolsMonitorsRootkit detectorsSetting up a testing environmentSetting up the debuggerStopping at the driver's entry pointLoading the driverRestoring the debugging stateSummary
Section 3: Examining Cross-Platform Malware
Handling Exploits and Shellcode
Getting familiar with vulnerabilities and exploitsTypes of vulnerabilitiesStack overflow vulnerabilityHeap overflow vulnerabilitiesThe use-after-free vulnerabilityLogical vulnerabilitiesTypes of exploitsCracking the shellcodeWhat's shellcode?Linux shellcode in x86-64Getting the absolute addressNull-free shellcodeLocal shell shellcodeReverse shell shellcodeLinux shellcode for ARMNull-free shellcodeWindows shellcodeGetting the Kernel32.dll's ImageBaseGetting the required APIs from Kernel32.dllThe download and execute shellcodeStatic and dynamic analysis of exploitsAnalysis workflowShellcode analysisExploring bypasses for exploit mitigation technologiesData execution prevention (DEP/NX)Return-oriented programmingAddress space layout randomizationDEP and partial ASLRDEP and full ASLR – partial ROP and chaining multiple vulnerabilitiesDEP and full ASLR – heap spray techniqueOther mitigation technologiesAnalyzing Microsoft Office exploitsFile structuresCompound file binary formatRich text formatOffice open XML formatStatic and dynamic analysis of MS Office exploitsStatic analysisDynamic analysisStudying malicious PDFsFile structureStatic and dynamic analysis of PDF filesStatic analysisDynamic analysisSummary
Reversing Bytecode Languages: .NET, Java, and More
The basic theory of bytecode languagesObject-oriented programmingInheritancePolymorphism.NET explained.NET file structure.NET COR20 headerMetadata streamsHow to identify a .NET application from PE characteristicsThe CIL language instruction setPushing into stack instructionsPulling out a value from the stackMathematical and logical operationsBranching instructionsCIL language to higher-level languagesLocal variable assignmentsLocal variable assignment with a method return valueBasic branching statementsLoops statements.NET malware analysis.NET analysis toolsStatic and dynamic analysis (with Dnspy).NET static analysis.NET dynamic analysisPatching a .NET sampleDealing with obfuscationObfuscated names for classes, methods, and othersEncrypted strings inside the binaryThe sample is obfuscated using an obfuscatorThe essentials of Visual BasicFile structureP-code versus native codeCommon p-code instructionsDissecting Visual Basic samplesStatic analysisP-codeNative codeDynamic analysisP-codeNative codeThe internals of Java samplesFile structureJVM instructionsStatic analysisDynamic analysisDealing with anti-reverse engineering solutionsPython—script language internalsFile structureBytecode instructionsAnalyzing compiled PythonStatic analysisDynamic analysisSummary
Scripts and Macros: Reversing, Deobfuscation, and Debugging
Classic shell script languagesWindows batch scriptingBashVBScript explainedBasic syntaxStatic and dynamic analysisDeobfuscationThose evil macros inside documentsBasic syntaxStatic and dynamic analysisBesides macrosThe power of PowerShellBasic syntaxStatic and dynamic analysisHandling JavaScriptBasic syntaxStatic and dynamic analysisAnti-reverse engineering tricksBehind C and C—even malware has its own backendThings to focus onStatic and dynamic analysisOther script languagesWhere to start fromQuestions to answerSummary
Section 4: Looking into IoT and Other Platforms
Dissecting Linux and IoT Malware
Explaining ELF files ELF structureSystem callsFilesystemNetworkProcess managementOtherSyscalls in assemblyCommon anti-reverse engineering tricksExploring common behavioral patternsInitial delivery and lateral movementPersistencePrivilege escalationInteraction with the command and control serverAttacking stageStatic and dynamic analysis of x86 (32- and 64-bit) samplesStatic analysisFile type detectorsData carvingDisassemblersActual toolsEnginesHow to chooseDynamic analysisTracersNetwork monitorsDebuggersBinary emulatorsRadare2 cheat sheetAnti-reverse engineering techniquesLearning Mirai, its clones, and moreHigh-level functionalityPropagationWeaponrySelf-defenseLater derivativesOther widespread familiesStatic and dynamic analysis of RISC samplesARMMIPSPowerPCSuperHSPARCHandling other architecturesWhat to start fromSummary
Introduction to macOS and iOS Threats
Understanding the role of the security modelmacOSSecurity policiesFilesystem hierarchy and encryptionDirectory structureEncryptionApps protectionGatekeeperApp sandboxOther technologiesiOSSystem securityData encryption and password managementApps' securityFile formats and APIsMach-OThinFatApplication bundles (.app)Info.plistmacOSiOSInstaller packages (.pkg)Apple disk images (.dmg)iOS app store packages (.ipa)APIsStatic and dynamic analyses of macOS and iOS samplesStatic analysisRetrieving samplesDisassemblers and decompilersAuxiliary tools and librariesDynamic and behavioral analysismacOSDebuggersMonitoring and dynamic instrumentationNetwork analysisiOSInstallers and loadersDebuggersDumping and decryptionMonitors and in-memory patchingNetwork analysisAttack stagesJailbreaks on demandPenetrationDeployment and persistencemacOSiOSAction phasemacOSiOSOther attack techniquesmacOSiOSAdvanced techniquesAnti-reverse-engineering (RE) tricksMisusing dynamic data exchange (DDE)User hidingUse of AppleScriptAPI hijackingRootkits for Mac—do they exist?Analysis workflowSummary
Analyzing Android Malware Samples
(Ab)using Android internals File hierarchyAndroid security modelProcess managementFilesystemApp permissionsSecurity servicesConsoleTo root or not to root?Understanding Dalvik and ART Dalvik VM (DVM)Android runtime (ART)APIsFile formatsDEXODEXOATVDEXARTELFAPKBytecode setMalware behavior patternsAttack stagesPenetrationDeploymentAction phaseAdvanced techniques—investment pays offPatching system librariesKeyloggingSelf-defenseRootkits—get it coveredStatic and dynamic analysis of threatsStatic analysisDisassembling and data extractionDecompilingDynamic analysisAndroid debug bridgeEmulatorsBehavioral analysis and tracingDebuggersAnalysis workflowSummary
Other Books You May Enjoy
Leave a review - let other readers know what you think

Content preview from Mastering Malware Analysis

Section 1: Fundamental Theory

In this section, you will be introduced to the core concepts required to successfully perform the static analysis of samples for various platforms, including the basics of architectures and assembly. While you may already have some prior knowledge of the x86 family, less common architectures, such as PowerPC or SH-4, are also extensively targeted by malware nowadays, so they shouldn't be underestimated. The following chapter is included in this section:

Chapter 1, A Crash Course in CISC/RISC and Programming Basics

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Mastering Malware Analysis - Second Edition

Publisher Resources

ISBN: 9781789610789

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Mastering Malware Analysis

by Alexey Kleymenov, Amr Thabet

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.