book

Mastering Malware Analysis - Second Edition

by Alexey Kleymenov, Amr Thabet

September 2022

Beginner

572 pages

14h 5m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorsAbout the reviewer
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Why malware analysis?Malware analysis in collecting threat intelligenceMalware analysis in incident responseMalware analysis in threat huntingMalware analysis in creating detectionsExploring types of malwareA short history of malware developmentMalware categoriesNaming conventionsThe MITRE ATT&CK framework explainedBasic terminologyEnterprise MatrixAPT and zero-day attacks and fileless malwareAPT attackZero-day attackFileless malwareChoosing your analysis strategyUnderstand your audienceAnswer your audience’s questionsDefine your goalsAvoid unnecessary technical detailsExample structuresTypical analysis workflowSetting up the environmentChoosing the virtualization softwareSafety featuresSummary
Basics of informaticsNumeral systemsBasic data units and data typesBitwise operationsArchitectures and their assemblyRegistersMemoryInstructions (CISC and RISC)Becoming familiar with x86 (IA-32 and x64)RegistersThe instruction structureThe instruction setArguments, local variables, and calling conventions (in x86 and x64)Exploring ARM assemblyBasicsInstruction setsBasics of MIPSBasicsThe instruction setDiving deep into PowerPCBasicsThe instruction setCovering the SuperH assemblyBasicsThe instruction setWorking with SPARCBasicsThe instruction setMoving from assembly to high-level programming languagesArithmetic statementsIf conditionsWhile loop conditionsSummary
Working with the PE header structureWhy PE?Exploring PE’s structurePE+ (x64 PE)PE header analysis toolsStatic and dynamic linkingStatic linkingDynamic linkingDynamic link librariesApplication programming interface (API)Using PE header information for static analysisHow to use the PE header for incident handlingHow to use a PE header for threat huntingPE loading and process creationBasic terminologyProcess creation step by stepPE file loading step by stepWOW64 processesBasics of dynamic analysis using OllyDbg and x64dbgDebugging toolsHow to analyze a sample with OllyDbgTypes of breakpointsModifying the program’s executionList strings, APIs, and cross-referencesSetting labels and commentsDifferences between OllyDbg and x64dbgDebugging malicious servicesWhat is a service?Attaching to servicesEssentials of behavioral analysisFile operationsRegistry operationsProcess operationsWinAPIsNetwork activitySandboxesSummary
Exploring packersExploring packing and encrypting toolsIdentifying a packed sampleTechnique 1 – using static signaturesTechnique 2 – evaluating PE section namesTechnique 3 – using stub execution signsTechnique 4 – detecting a small import tableAutomatically unpacking packed samplesTechnique 1 – the official unpacking processTechnique 2 – using OllyScript with OllyDbgTechnique 3 – using generic unpackersTechnique 4 – emulationTechnique 5 – memory dumpsManual unpacking techniquesTechnique 1 – memory breakpoint on executionTechnique 2 – call stack backtracingTechnique 3 – monitoring memory allocated spaces for unpacked codeTechnique 4 – in-place unpackingTechnique 5 – searching for and transferring control to OEPTechnique 6 – stack restoration-basedDumping the unpacked sample and fixing the import tableDumping the processFixing the import tableIdentifying simple encryption algorithms and functionsTypes of encryption algorithmsBasic encryption algorithmsIdentifying encryption functions in disassemblyString search detection techniques for simple algorithmsIdentifying the RC4 encryption algorithmAdvanced symmetric and asymmetric encryption algorithmsExtracting information from Windows cryptography APIsCryptography API: Next Generation (CNG)Applications of encryption in modern malware – Vawtrak banking TrojanString and API name encryptionNetwork communication encryptionUsing IDA for decryption and unpackingIDA tips and tricksClassic and new syntax of IDA scriptsDynamic string decryptionDynamic WinAPIs resolutionSummary
Understanding process injectionWhat’s process injection?Why process injection?DLL injectionWindows-supported DLL injectionA simple DLL injection techniqueDiving deeper into process injectionFinding the victim processCode block injectionReflective DLL injectionStuxnet secret technique – process hollowingA dynamic analysis of code injectionTechnique 1 – Debug it where it isTechnique 2 – Attach to the targeted processTechnique 3 – Dealing with process hollowingMemory forensics techniques for process injectionTechnique 1 – Detecting code injection and reflective DLL injectionTechnique 2 – Detecting process hollowingTechnique 3 – Detecting process hollowing using the HollowFind pluginUnderstanding API hookingWhy API hooking?Working with API hookingDetecting API hooking using memory forensicsExploring IAT hookingSummary
Exploring debugger detectionUsing PEB informationUsing EPROCESS informationUsing DebugObjectUsing handlesUsing exceptionsUsing parent processesHandling the evasion of debugger breakpointsDetecting software breakpoints (INT3)Detecting single-stepping breakpoints using a trap flagDetecting single-stepping using timing techniquesEvading hardware breakpointsMemory breakpointsEscaping the debuggerProcess injectionTLS callbacksWindows events callbacksAttacking the debuggerUnderstanding obfuscation and anti-disassemblersEncryptionJunk codeCode transportationDynamic API calling with checksumProxy functions and proxy argument stackingUsing the COM functionalityDetecting and evading behavioral analysis toolsFinding the tool processSearching for the tool windowDetecting sandboxes and VMsDifferent output between VMs and real machinesDetecting virtualization processes and servicesDetecting virtualization through registry keysDetecting VMs using WMIOther VM detection techniquesDetecting sandboxes using default settingsSummary

Kernel mode versus user modeProtection ringsWindows internalsThe anatomy of WindowsThe execution path from user mode to kernel modeRootkits and device driversWhat is a rootkit?Types of rootkitsWhat is a device driver?Hooking mechanismsHooking the SYSENTER entry functionModifying SSDT in an x86 environmentModifying SSDT in an x64 environmentPatching SSDT functionsIRP hookingDKOMThe kernel objects – EPROCESS and ETHREADHow do rootkits perform an object manipulation attack?Process injection in kernel modeExecuting the inject code using APC queuingKPP in x64 systems (PatchGuard)Bypassing driver signature enforcementBypassing PatchGuard – the Turla exampleBypassing PatchGuard – GhostHookStatic and dynamic analysis in kernel modeStatic analysisDynamic and behavioral analysisSetting up a testing environmentSetting up the debuggerStopping at the driver's entry pointLoading the driverRestoring the debugging stateSummary
Getting familiar with vulnerabilities and exploitsTypes of vulnerabilitiesTypes of exploitsCracking the shellcodeWhat’s shellcode?Linux shellcode in x86-64Linux shellcode for ARMWindows shellcodeStatic and dynamic analysis of exploitsExploring bypasses for exploit mitigation technologiesData execution prevention (DEP/NX)Return-oriented programmingAddress space layout randomizationOther mitigation technologiesAnalyzing Microsoft Office exploitsFile structuresStatic and dynamic analysis of MS Office exploitsStudying malicious PDFsFile structureStatic and dynamic analysis of PDF filesSummary
The basic theory of bytecode languagesObject-oriented programmingInheritancePolymorphism.NET explained.NET file structureHow to identify a .NET application from PE characteristicsThe CIL language instruction setCIL language into higher-level languages.NET malware analysis.NET analysis toolsStatic and dynamic analysisDealing with obfuscationThe essentials of Visual BasicFile structureP-code versus native codeCommon p-code instructionsDissecting Visual Basic samplesStatic analysisDynamic analysisThe internals of Java samplesFile structureJVM instructionsStatic analysisDynamic analysisDealing with anti-reverse engineering solutionsAnalyzing compiled Python threatsFile structureBytecode instructionsStatic analysisDynamic analysisSummary
Classic shell script languagesWindows batch scriptingBashVBScript explainedBasic syntaxStatic and dynamic analysisDeobfuscationVBA and Excel 4.0 (XLM) macros and moreVBA macrosExcel 4.0 (XLM) macrosBesides macrosThe power of PowerShellBasic syntaxObfuscationStatic and dynamic analysisHandling JavaScriptBasic syntaxAnti-reverse engineering tricksStatic and dynamic analysisBehind C&C – even malware has its own backendThings to focus onStatic and dynamic analysisOther script languagesWhere to startQuestions to answerSummary
Explaining ELF filesThe ELF structureSystem callsExploring common behavioral patternsInitial access and lateral movementPersistencePrivilege escalationCommand and controlImpactDefense evasionStatic and dynamic analysis of x86 (32- and 64-bit) samplesStatic analysisDynamic analysisA radare2 cheat sheetLearning about Mirai, its clones, and moreHigh-level functionalityLater derivativesOther widespread familiesStatic and dynamic analysis of RISC samplesARMMIPSPowerPCSuperHSPARCHandling other architecturesWhat to start fromSummary
Understanding the role of the security modelmacOSOther technologiesiOSFile formats and APIsMach-OApplication bundles (.app)Installer packages (.pkg)Apple disk images (.dmg)iOS app store packages (.ipa)APIsAttack stagesJailbreaks on demandInitial accessExecution and persistenceImpactOther attack techniquesAdvanced techniquesAnti-analysis and detection tricksMisusing dynamic data exchange (DDE)User hidingUsing AppleScriptAPI hijackingOther techniquesRootkits for Mac – do they exist?Static and dynamic analysis of macOS and iOS samplesStatic analysisDynamic and behavioral analysisThe analysis workflowSummary
(Ab)using the Android internalsThe file hierarchyThe Android security modelTo root or not to root?Understanding Dalvik and ARTDalvik VM (DVM)Android runtime (ART)The bytecode setFile formats and APIsDEXODEXOATVDEXARTELFAPKAPIsMalware behavior patternsInitial accessPrivilege escalationPersistenceImpactCollectionDefence evasionStatic and dynamic analysis of threatsStatic analysisDynamic analysisBehavioral analysis and tracingThe analysis workflowSummary
Other Books You May EnjoyPackt is searching for authors like you

Content preview from Mastering Malware Analysis - Second Edition

5 Inspecting Process Injection and API Hooking

In this chapter, we are going to explore more advanced techniques that are used by malware authors for various reasons, including bypassing firewalls, tricking reverse engineers, and monitoring and collecting user information in order to steal credit card data and for other purposes.

We will be diving into various process injection techniques, including DLL injection and process hollowing (an advanced technique that was introduced by Stuxnet), and explain how to deal with them. Later, we will look at API hooking, IAT hooking, and other hooking techniques that are used by malware authors and how to handle them.

By the end of this chapter, you will have extended your knowledge of the Windows platform ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Publisher Resources

ISBN: 9781803240244Supplemental Content

Mastering Malware Analysis - Second Edition

by Alexey Kleymenov, Amr Thabet

5

Inspecting Process Injection and API Hooking

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

You might also like