book

Mastering Malware Analysis

by Alexey Kleymenov, Amr Thabet

June 2019

Beginner

562 pages

11h 36m

English

Packt Publishing

Read now

Unlock full access

Mastering Malware Analysis
Why subscribe?
About the authorsAbout the reviewersPackt is searching for authors like you
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchReviews
Basic conceptsRegistersMemoryVirtual memoryStackBranches, loops, and conditionsExceptions, interrupts, and communicating with other devicesAssembly languagesCISC versus RISCTypes of instructionsBecoming familiar with x86 (IA-32 and x64)RegistersSpecial registersThe instruction structureopcodedestsrcThe instruction setData manipulation instructionsData transfer instructionsFlow control instructionsArguments, local variables, and calling conventions (in x86 and x64)stdcallArgumentsLocal variablescdeclfastcallthiscallThe x64 calling conventionExploring ARM assemblyBasicsInstruction setsBasics of MIPSBasicsThe instruction setDiving deep into PowerPCBasicsThe instruction setCovering the SuperH assemblyBasicsThe instruction setWorking with SPARCBasicsThe instruction setMoving from assembly to high-level programming languagesArithmetic statementsIf conditionsWhile loop conditionsSummary
Working with the PE header structureWhy PE?Exploring PE structureMZ headerPE headerFile headerOptional headerData directorySection tablePE+ (x64 PE)PE header analysis toolsStatic and dynamic linkingStatic linkingDynamic linkingDynamic link librariesApplication programming interfaceDynamic API loadingUsing PE header information for static analysisHow to use PE header for incident handlingHow to use a PE header for threat intelligencePE loading and process creationBasic terminologyWhat's process?Virtual memory to physical memory mappingThreadsImportant data structures: TIB, TEB, and PEBProcess loading step by stepPE file loading step by stepWOW64 processesDynamic analysis with OllyDbg/Immunity DebuggerDebugging toolsHow to analyze a sample with OllyDbgTypes of breakpointsStep into/step over breakpointINT3 breakpointMemory breakpointsHardware breakpointsModifying the program executionPatching—modifying the program's assembly instructionsChange EFlagsModifying the instruction pointer valueChanging the program dataDebugging malicious servicesWhat is service?Attaching to the serviceSummary
Exploring packersExploring packing and encrypting toolsIdentifying a packed sampleTechnique 1 – checking PE tool static signaturesTechnique 2 – evaluating PE section namesTechnique 3 – using stub execution signsTechnique 4 – detecting a small import tableAutomatically unpacking packed samplesTechnique 1 – the official unpacking processTechnique 2 – using OllyScript with OllyDbgTechnique 3 – using generic unpackersTechnique 4 – emulationTechnique 5 – memory dumpsManual unpacking using OllyDbgTechnique 6 – memory breakpoint on executionStep 1 – setting the breakpointsStep 2 – turning on Data Execution PreventionStep 3 – preventing any further attempts to change memory permissionsStep 4 – executing and getting the OEPTechnique 7 – call stack backtracingStep 1 – setting the breakpointsStep 2 – following the call stackStep 3 – reaching the OEPTechnique 8 – monitoring memory allocated spaces for unpacked codeTechnique 9 – in-place unpackingTechnique 10 – stack restoration-basedDumping the unpacked sample and fixing the import tableDumping the processFixing the import tableIdentifying different encryption algorithms and functionsTypes of encryption algorithmsBasic encryption algorithmsHow to identify encryption functionsString search detection techniques for simple algorithmsThe basics of X-RAYINGSimple static encryptionOther encryption algorithmsX-RAYING tools for malware analysis and detectionIdentifying the RC4 encryption algorithmThe RC4 encryption algorithmKey-scheduling algorithmPseudo-random generation algorithmIdentifying RC4 algorithms in a malware sampleStandard symmetric and asymmetric encryption algorithmsExtracting information from Windows cryptography APIsStep 1 – initializing and connecting to the cryptographic service provider (CSP)Step 2 – preparing the keyStep 3 – encrypting or decrypting the dataStep 4 – freeing the memoryCryptography API next generation (CNG)Applications of encryption in modern malware – Vawtrak banking TrojanString and API name encryptionNetwork communication encryptionUsing IDA for decryption and unpackingIDA tips and tricksStatic analysisDynamic analysisClassic and new syntax of IDA scriptsDynamic string decryptionDynamic WinAPIs resolutionSummary

Understanding process injectionWhat's process injection?Why process injection?DLL injectionWindows-supported DLL injectionA simple DLL injection techniqueWorking with process injectionGetting the list of running processesCode injectionAdvanced code injection-reflective DLL injectionStuxnet secret technique-process hollowingDynamic analysis of code injectionTechnique 1—debug it where it isTechnique 2—attach to the targeted processTechnique 3—dealing with process hollowingMemory forensics techniques for process injectionTechnique 1—detecting code injection and reflective DLL injection Technique 2—detecting process hollowingTechnique 3—detecting process hollowing using the HollowFind pluginUnderstanding API hookingWhy API hooking?Working with API hookingInline API hookingInline API hooking with trampolineInline API hooking with a length disassemblerDetecting API hooking using memory forensicsExploring IAT hookingSummary
Exploring debugger detectionDirect check for debugger presenceDetecting a debugger through an environment changeDetecting a debugger using parent processesHandling debugger breakpoints evasionDetecting software breakpoints (INT3)Detecting single-stepping breakpoints (trap flag)Detecting a trap flag using the SS registerDetecting single-stepping using timing techniquesEvading hardware breakpointsWhat is structured exception handling?Detecting and removing hardware breakpointsMemory breakpointsEscaping the debuggerProcess injectionTLS callbacksWindows events callbacksObfuscation and anti-disassemblersEncryptionJunk code insertionCode transportationDynamic API calling with checksumProxy functions and proxy argument stackingDetecting and evading behavioral analysis toolsFinding the tool processSearching for the tool windowDetecting sandboxes and virtual machinesDifferent output between virtual machines and real machinesDetecting virtualization processes and servicesDetecting virtualization through registry keysDetecting virtual machines using PowerShellDetecting sandboxes by using default settingsOther techniquesSummary
Kernel mode versus user modeProtection ringsWindows internalsThe infrastructure of WindowsThe execution path from user mode to kernel modeRootkits and device driversWhat is a rootkit?Types of rootkitsWhat is a device driver?Hooking mechanismsSSDT hookingHooking the SYSENTER entry functionModifying SSDT in an x86 environmentModifying SSDT in an x64 environmentHooking SSDT functionsIRP hookingDevices and major functionsAttaching to a deviceModifying the IRP response and setting a completion routineDKOMThe kernel objects—EPROCESS and ETHREADHow do rootkits perform an object manipulation attack?Process injection in kernel modeExecuting the inject code using APC queuingKPP in x64 systems (PatchGuard)Bypassing driver signature enforcementBypassing PatchGuard—the Turla exampleBypassing PatchGuard—GhostHookDisabling PatchGuard using the Command PromptStatic and dynamic analysis in kernel modeStatic analysisToolsTips and tricksDynamic and behavioral analysisToolsMonitorsRootkit detectorsSetting up a testing environmentSetting up the debuggerStopping at the driver's entry pointLoading the driverRestoring the debugging stateSummary
Getting familiar with vulnerabilities and exploitsTypes of vulnerabilitiesStack overflow vulnerabilityHeap overflow vulnerabilitiesThe use-after-free vulnerabilityLogical vulnerabilitiesTypes of exploitsCracking the shellcodeWhat's shellcode?Linux shellcode in x86-64Getting the absolute addressNull-free shellcodeLocal shell shellcodeReverse shell shellcodeLinux shellcode for ARMNull-free shellcodeWindows shellcodeGetting the Kernel32.dll's ImageBaseGetting the required APIs from Kernel32.dllThe download and execute shellcodeStatic and dynamic analysis of exploitsAnalysis workflowShellcode analysisExploring bypasses for exploit mitigation technologiesData execution prevention (DEP/NX)Return-oriented programmingAddress space layout randomizationDEP and partial ASLRDEP and full ASLR – partial ROP and chaining multiple vulnerabilitiesDEP and full ASLR – heap spray techniqueOther mitigation technologiesAnalyzing Microsoft Office exploitsFile structuresCompound file binary formatRich text formatOffice open XML formatStatic and dynamic analysis of MS Office exploitsStatic analysisDynamic analysisStudying malicious PDFsFile structureStatic and dynamic analysis of PDF filesStatic analysisDynamic analysisSummary
The basic theory of bytecode languagesObject-oriented programmingInheritancePolymorphism.NET explained.NET file structure.NET COR20 headerMetadata streamsHow to identify a .NET application from PE characteristicsThe CIL language instruction setPushing into stack instructionsPulling out a value from the stackMathematical and logical operationsBranching instructionsCIL language to higher-level languagesLocal variable assignmentsLocal variable assignment with a method return valueBasic branching statementsLoops statements.NET malware analysis.NET analysis toolsStatic and dynamic analysis (with Dnspy).NET static analysis.NET dynamic analysisPatching a .NET sampleDealing with obfuscationObfuscated names for classes, methods, and othersEncrypted strings inside the binaryThe sample is obfuscated using an obfuscatorThe essentials of Visual BasicFile structureP-code versus native codeCommon p-code instructionsDissecting Visual Basic samplesStatic analysisP-codeNative codeDynamic analysisP-codeNative codeThe internals of Java samplesFile structureJVM instructionsStatic analysisDynamic analysisDealing with anti-reverse engineering solutionsPython—script language internalsFile structureBytecode instructionsAnalyzing compiled PythonStatic analysisDynamic analysisSummary
Classic shell script languagesWindows batch scriptingBashVBScript explainedBasic syntaxStatic and dynamic analysisDeobfuscationThose evil macros inside documentsBasic syntaxStatic and dynamic analysisBesides macrosThe power of PowerShellBasic syntaxStatic and dynamic analysisHandling JavaScriptBasic syntaxStatic and dynamic analysisAnti-reverse engineering tricksBehind C and C—even malware has its own backendThings to focus onStatic and dynamic analysisOther script languagesWhere to start fromQuestions to answerSummary
Explaining ELF files ELF structureSystem callsFilesystemNetworkProcess managementOtherSyscalls in assemblyCommon anti-reverse engineering tricksExploring common behavioral patternsInitial delivery and lateral movementPersistencePrivilege escalationInteraction with the command and control serverAttacking stageStatic and dynamic analysis of x86 (32- and 64-bit) samplesStatic analysisFile type detectorsData carvingDisassemblersActual toolsEnginesHow to chooseDynamic analysisTracersNetwork monitorsDebuggersBinary emulatorsRadare2 cheat sheetAnti-reverse engineering techniquesLearning Mirai, its clones, and moreHigh-level functionalityPropagationWeaponrySelf-defenseLater derivativesOther widespread familiesStatic and dynamic analysis of RISC samplesARMMIPSPowerPCSuperHSPARCHandling other architecturesWhat to start fromSummary
Understanding the role of the security modelmacOSSecurity policiesFilesystem hierarchy and encryptionDirectory structureEncryptionApps protectionGatekeeperApp sandboxOther technologiesiOSSystem securityData encryption and password managementApps' securityFile formats and APIsMach-OThinFatApplication bundles (.app)Info.plistmacOSiOSInstaller packages (.pkg)Apple disk images (.dmg)iOS app store packages (.ipa)APIsStatic and dynamic analyses of macOS and iOS samplesStatic analysisRetrieving samplesDisassemblers and decompilersAuxiliary tools and librariesDynamic and behavioral analysismacOSDebuggersMonitoring and dynamic instrumentationNetwork analysisiOSInstallers and loadersDebuggersDumping and decryptionMonitors and in-memory patchingNetwork analysisAttack stagesJailbreaks on demandPenetrationDeployment and persistencemacOSiOSAction phasemacOSiOSOther attack techniquesmacOSiOSAdvanced techniquesAnti-reverse-engineering (RE) tricksMisusing dynamic data exchange (DDE)User hidingUse of AppleScriptAPI hijackingRootkits for Mac—do they exist?Analysis workflowSummary
(Ab)using Android internals File hierarchyAndroid security modelProcess managementFilesystemApp permissionsSecurity servicesConsoleTo root or not to root?Understanding Dalvik and ART Dalvik VM (DVM)Android runtime (ART)APIsFile formatsDEXODEXOATVDEXARTELFAPKBytecode setMalware behavior patternsAttack stagesPenetrationDeploymentAction phaseAdvanced techniques—investment pays offPatching system librariesKeyloggingSelf-defenseRootkits—get it coveredStatic and dynamic analysis of threatsStatic analysisDisassembling and data extractionDecompilingDynamic analysisAndroid debug bridgeEmulatorsBehavioral analysis and tracingDebuggersAnalysis workflowSummary
Leave a review - let other readers know what you think

Content preview from Mastering Malware Analysis

Reversing Bytecode Languages: .NET, Java, and More

The beauty of cross-platform compiled programs is in their flexibility as you don't need to port each program to different systems. In this chapter, we will take a look at how malware authors are trying to leverage these advantages for evil purposes. In addition, you will be provided with an arsenal of tools and techniques whose aim is to make analysis quick and efficient.

This chapter is divided into the following sections to facilitate the learning process:

The basic theory of bytecode languages
.NET explained
.NET malware analysis
The essentials of Visual Basic
Dissecting Visual Basic samples
The internals of Java samples
Python—script language internals
Analyzing compiled Python

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Mastering Malware Analysis - Second Edition

Alexey Kleymenov, Amr Thabet

Advanced Malware Analysis

Christopher Elisan

Advanced Malware Analysis

Munir Njenga

Learning Malware Analysis

Monnappa K A

Publisher Resources

ISBN: 9781789610789Supplemental Content

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,and much more.

You might also like

Mastering Malware Analysis - Second Edition

Advanced Malware Analysis

Advanced Malware Analysis

Learning Malware Analysis

Publisher Resources

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.