book

Hacks, Leaks, and Revelations

Name: Hacks, Leaks, and Revelations
Author: Micah Lee
ISBN: 9781718503120

by Micah Lee

January 2024

Intermediate to advanced

544 pages

15h 21m

English

No Starch Press

Read now

Unlock full access

Praise for Hacks, Leaks, and Revelations
Title Page
Copyright
Dedication
About the Author and Technical Reviewer
Acknowledgments
Introduction
Why I Wrote This BookWhat You’ll LearnWhat You’ll Need
Part I: Sources and Datasets
1. Protecting Sources and Yourself
Safely Communicating with SourcesWorking with Public DataProtecting Sensitive InformationMinimizing the Digital TrailWorking with Hackers and WhistleblowersSecure Storage for DatasetsLow-Sensitivity DatasetsMedium-Sensitivity DatasetsHigh-Sensitivity DatasetsAuthenticating DatasetsThe AFLDS DatasetThe WikiLeaks Twitter Group ChatRedactionWhat Data to PublishWhat to RedactMaking Requests for CommentPassword ManagersDisk EncryptionExercise 1-1: Encrypt Your Internal DiskWindowsmacOSLinuxExercise 1-2: Encrypt a USB DiskWindowsmacOSLinuxProtecting Yourself from Malicious DocumentsExercise 1-3: Install and Use DangerzoneSummary
2. Acquiring Datasets
The End of WikiLeaksDistributed Denial of SecretsDownloading Datasets with BitTorrentThe Origins of BlueLeaksExercise 2-1: Download the BlueLeaks DatasetCommunicating with Encrypted Messaging AppsExercise 2-2: Install and Practice Using SignalEncrypting Messages with PGPStaying Anonymous Online with Tor and OnionShareExercise 2-3: Play with Tor and OnionShareCommunicating with My Tea Party Patriots SourceOther Options for Acquiring Datasets from SourcesEncrypted USB DrivesVirtual Private ServersWhistleblower Submission SystemsSummary

Part II: Tools of the Trade
3. The Command Line Interface
Introducing the Command LineThe ShellUsers and PathsUser PrivilegesExercise 3-1: Install Ubuntu in WindowsBasic Command Line UsageOpening a TerminalClearing Your Screen and Exiting the ShellExploring Files and DirectoriesNavigating Relative and Absolute PathsChanging DirectoriesUsing the help ArgumentAccessing Man PagesTips for Navigating the TerminalEntering Commands with Tab CompletionEditing CommandsDealing with Spaces in FilenamesUsing Single Quotes Around Double QuotesInstalling and Uninstalling Software with Package ManagersExercise 3-2: Manage Packages with Homebrew on macOSExercise 3-3: Manage Packages with apt on Windows or LinuxExercise 3-4: Practice Using the Command Line with cURLDownload a Web Page with cURLSave a Web Page to a FileText Files vs. Binary FilesExercise 3-5: Install the VS Code Text EditorExercise 3-6: Write Your First Shell ScriptNavigate to Your USB DiskCreate an Exercises FolderOpen a VS Code WorkspaceWrite the Shell ScriptRun the Shell ScriptExercise 3-7: Clone the Book’s GitHub RepositorySummary
4. Exploring Datasets in the Terminal
Introducing for LoopsExercise 4-1: Unzip the BlueLeaks DatasetUnzip Files on macOS or LinuxUnzip Files on WindowsOrganize Your FilesHow the Hacker Obtained the BlueLeaks DataExercise 4-2: Explore BlueLeaks on the Command LineCalculate How Much Disk Space Folders UseUse Pipes and Sort OutputCreate an Inventory of Filenames in a DatasetCount the Files in a DatasetExercise 4-3: Find Revelations in BlueLeaks with grepFilter for Documents Mentioning AntifaFilter for Certain Types of FilesUse grep with Regular ExpressionsSearch Files in Bulk with grepEncrypted Data in the BlueLeaks DatasetData Analysis with Servers in the CloudExercise 4-4: Set Up a VPSGenerate an SSH KeyAdd Your Public Key to the Cloud ProviderCreate a VPSSSH into Your ServerStart a Byobu SessionInstall UpdatesExercise 4-5: Explore the Oath Keepers Dataset RemotelySummary
5. Docker, Aleph, and Making Datasets Searchable
Introducing Docker and Linux ContainersExercise 5-1: Initialize Docker Desktop on Windows and macOSExercise 5-2: Initialize Docker Engine on LinuxRunning Containers with DockerRunning an Ubuntu ContainerListing and Killing ContainersMounting and Removing VolumesPassing Environment VariablesRunning Server SoftwareFreeing Up Disk SpaceExercise 5-3: Run a WordPress Site with Docker ComposeMake a docker-compose.yaml FileStart Your WordPress SiteIntroducing AlephExercise 5-4: Run Aleph Locally in Linux ContainersUsing Aleph’s Web and Command Line InterfacesIndexing Data in AlephExercise 5-5: Index a BlueLeaks Folder in AlephMount Your Datasets into the Aleph ShellIndex the icefishx FolderCheck Indexing StatusExplore BlueLeaks with AlephAdditional Aleph FeaturesDedicated Aleph ServersSummary
6. Reading Other People’s Email
The Email Protocol and Message StructureFile Formats for Email DumpsEML FilesMBOX FilesPST Outlook Data FilesExercise 6-1: Download Email Dumps from Three DatasetsThe Nauru Police Force DatasetThe Oath Keepers DatasetThe Heritage Foundation DatasetResearching Email Dumps with ThunderbirdExercise 6-2: Configure Thunderbird for Email DumpsReading Individual EML Files with ThunderbirdExercise 6-3: Import the Nauru Police Force EML Email DumpSearching Email in ThunderbirdQuick Filter SearchesThe Search Messages DialogExercise 6-4: Import the Oath Keepers MBOX Email DumpExercise 6-5: Import the Heritage Foundation PST Email DumpOther Tools for Researching Email DumpsMicrosoft OutlookAlephSummary
Part III: Python Programming
7. An Introduction to Python
Exercise 7-1: Install PythonWindowsLinuxmacOSExercise 7-2: Write Your First Python ScriptPython BasicsThe Interactive Python InterpreterCommentsMath with PythonStringsExercise 7-3: Write a Python Script with Variables, Math, and StringsLists and LoopsDefining and Printing ListsRunning for LoopsControl FlowComparison Operatorsif StatementsNested Code BlocksSearching ListsLogical OperatorsException HandlingExercise 7-4: Practice Loops and Control FlowFunctionsThe def KeywordDefault ArgumentsReturn ValuesDocstringsExercise 7-5: Practice Writing FunctionsSummary
8. Working with Data in Python
ModulesPython Script TemplateExercise 8-1: Traverse the Files in BlueLeaksList the Filenames in a FolderCount the Files and Folders in a FolderTraverse Folders with os.walk()Exercise 8-2: Find the Largest Files in BlueLeaksThird-Party ModulesExercise 8-3: Practice Command Line Arguments with ClickAvoiding Hardcoding with Command Line ArgumentsExercise 8-4: Find the Largest Files in Any DatasetDictionariesDefining DictionariesGetting and Setting ValuesNavigating Dictionaries and Lists in the Conti Chat LogsExploring Dictionaries and Lists Full of Data in PythonSelecting Values in Dictionaries and ListsAnalyzing Data Stored in Dictionaries and ListsExercise 8-5: Map Out the CSVs in BlueLeaksAccept a Command Line ArgumentLoop Through the BlueLeaks FoldersFill Up the DictionaryDisplay the OutputReading and Writing FilesOpening FilesWriting Lines to a FileReading Lines from a FileExercise 8-6: Practice Reading and Writing FilesSummary
Part IV: Structured Data
9. Blueleaks, Black Lives Matter, and the CSV File Format
Installing Spreadsheet SoftwareIntroducing the CSV File FormatExploring CSV Files with Spreadsheet Software and Text EditorsMy BlueLeaks InvestigationFocusing on a Fusion CenterIntroducing NCRICInvestigating a SARReading and Writing CSV Files in PythonExercise 9-1: Make BlueLeaks CSVs More ReadableAccept the CSV Path as an ArgumentLoop Through the CSV RowsDisplay CSV Fields on Separate LinesHow to Read Bulk Email from Fusion CentersLists of Black Lives Matter Demonstrations“Intelligence” Memos from the FBI and DHSA Brief HTML PrimerExercise 9-2: Make Bulk Email ReadableAccept the Command Line ArgumentsCreate the Output FolderDefine the Filename for Each RowWrite the HTML Version of Each Bulk EmailDiscovering the Names and URLs of BlueLeaks SitesExercise 9-3: Make a CSV of BlueLeaks SitesOpen a CSV for WritingFind All the Company.csv FilesAdd BlueLeaks Sites to the CSVSummary
10. Blueleaks Explorer
Undiscovered Revelations in BlueLeaksExercise 10-1: Install BlueLeaks ExplorerCreate the Docker Compose Configuration FileBring Up the ContainersInitialize the DatabasesThe Structure of NCRICExploring Tables and RelationshipsSearching for KeywordsBuilding Your Own BlueLeaks StructureDefining the JRIC StructureShowing Useful FieldsChanging Field TypesAdding JRIC’s Leads TableBuilding a RelationshipVerifying BlueLeaks DataExercise 10-2: Finish Building the Structure for JRICThe Technology Behind BlueLeaks ExplorerThe BackendThe FrontendSummary
11. Parler, the January 6 Insurrection, and the JSON File format
The Origins of the Parler DatasetHow the Parler Videos Were ArchivedThe Dataset’s Impact on Trump’s Second ImpeachmentExercise 11-1: Download and Extract Parler Video MetadataDownload the MetadataUncompress and Download Individual Parler VideosExtract Parler MetadataThe JSON File FormatUnderstanding JSON SyntaxParsing JSON with PythonHandling Exceptions with JSONTools for Exploring JSON DataCounting Videos with GPS Coordinates Using grepFormatting and Searching Data with the jq CommandExercise 11-2: Write a Script to Filter for Videos with GPS from January 6, 2021Accept the Parler Metadata Path as an ArgumentLoop Through Parler Metadata FilesFilter for Videos with GPS CoordinatesFilter for Videos from January 6, 2021Working with GPS CoordinatesSearching by Latitude and LongitudeConverting Between GPS Coordinate FormatsCalculating GPS Distance in PythonFinding the Center of Washington, DCExercise 11-3: Update the Script to Filter for Insurrection VideosPlotting GPS Coordinates on a Map with simplekmlExercise 11-4: Create KML Files to Visualize Location DataCreate a KML File for All Videos with GPS CoordinatesCreate KML Files for Videos from January 6, 2021Visualizing Location Data with Google EarthViewing Metadata with ExifToolSummary
12. Epik Fail, Extremism Research, and SQL Databases
The Structure of SQL DatabasesRelational DatabasesClients and ServersTables, Columns, and TypesExercise 12-1: Create and Test a MySQL Server Using Docker and AdminerRun the ServerConnect to the Database with AdminerCreate a Test DatabaseExercise 12-2: Query Your SQL DatabaseINSERT StatementsSELECT StatementsJOIN ClausesUPDATE StatementsDELETE StatementsIntroducing the MySQL Command Line ClientExercise 12-3: Install and Test the Command Line MySQL ClientMySQL-Specific QueriesThe History of EpikThe Epik HackEpik’s WHOIS DataExercise 12-4: Download and Extract Part of the Epik DatasetExercise 12-5: Import Epik Data into MySQLCreate a Database for api_systemImport api_system DataExploring Epik’s SQL DatabaseThe domain TableThe privacy TableThe hosting and hosting_server TablesWorking with Epik Data in the CloudSummary
Part V: Case Studies
13. Pandemic Profiteers and Covid-19 Disinformation
The Origins of AFLDSThe Cadence Health and Ravkoo DatasetsExtracting the Data into an Encrypted File ContainerAnalyzing the Data with Command Line ToolsCreating a Single Spreadsheet of PatientsCalculating Revenue from Prescriptions Filled by RavkooFinding the Price and Quantity of Drugs SoldCategorizing Prescription Data by DrugA Deeper Look at the Cadence Health Patient DataFinding Cadence’s PartnersSearching for Patients by CitySearching for Patients by AgeAuthenticating the DataThe AftermathHIPAA’s Breach Notification RuleCongressional InvestigationSimone Gold’s New Business VentureScandal and Infighting at AFLDSSummary
14. Neo-Nazis and their Chatrooms
How Antifascists Infiltrated Neo-Nazi Discord ServersAnalyzing Leaked Chat LogsMaking JSON Files ReadableExploring Objects, Keys, and Values with jqConverting TimestampsFinding UsernamesThe Discord History TrackerA Script to Search the JSON FilesMy Discord Analysis CodeDesigning the SQL DatabaseImporting Chat Logs into the SQL DatabaseBuilding the Web InterfaceUsing Discord Analysis to Find RevelationsThe Pony Power Discord ServerThe Launch of DiscordLeaksThe AftermathThe Lawsuit Against Unite the RightThe Patriot Front Chat LogsSummary
Afterword
A. Solutions to Common WSL Problems
Understanding WSL’s Linux FilesystemThe Disk Performance ProblemSolving the Disk Performance ProblemStoring Only Active Datasets in LinuxStoring Your Linux Filesystem on a USB DiskNext Steps
B. Scraping the Web
Legal ConsiderationsHTTP RequestsScraping TechniquesLoading Pages with HTTPXParsing HTML with Beautiful SoupAutomating Web Browsers with SeleniumNext Steps
Index

Content preview from Hacks, Leaks, and Revelations

`9` `BLUELEAKS, BLACK LIVES MATTER, AND THE CSV FILE FORMAT`

The BlueLeaks dataset is full of an overwhelming number of documents, but it’s not immediately obvious where to start or how to make sense of the data they contain. Before beginning an investigation, I needed a way to efficiently determine the significance of these documents. After manually digging through many files, I discovered that the context I needed was in the hundreds of CSV spreadsheets in each BlueLeaks folder. In this chapter, you’ll learn how to investigate CSV files like these yourself.

You’ll view CSVs in both graphical spreadsheet and text editing software, write Python code to loop through the rows of a CSV, and save CSVs of your own. You’ll then put this knowledge ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098168773Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Hacks, Leaks, and Revelations

by Micah Lee

`9` `BLUELEAKS, BLACK LIVES MATTER, AND THE CSV FILE FORMAT`

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.