book

Data Engineering for Cybersecurity

Name: Data Engineering for Cybersecurity
Author: James Bonifield
ISBN: 9781718504028

by James Bonifield

August 2025

Intermediate to advanced

344 pages

8h 34m

English

No Starch Press

Read now

Unlock full access

Title Page
Copyright
Dedication
About the Author and the Technical Reviewers
Acknowledgments
Introduction
Who This Book Is ForWhat’s in This BookPrerequisitesOnline Resources
Part I: Foundations of Secure Data Engineering
1. Data Engineering Basics
Common Data Engineering TasksGetting Data to the SIEMManaging Throughput and LatencyEnriching and Standardizing EventsExample ArchitecturesA Basic Data PipelineProactive Data RetrievalTemporary Data CentralizationEvent CachingData Serialization FormatsJSONYAMLElastic Common SchemaVirtual Machine SetupWindows File Transfer and Shell Access ToolsOther Useful ToolsSummary
2. Network Encryption
TLS ComponentsInteractionsMutual TLSGenerating a Certificate AuthorityRoot ConfigurationRoot GenerationIntermediate ConfigurationIntermediate SigningChain FilesSetting Up TLS for LogstashFlex ConfigurationServer Keys and CertificatesDirectory PreparationLogstash ConfigurationFirewall PortsSSHEnabling SSHCreating SSH KeysStarting the AgentCreating Configuration FilesDistributing Public KeysVerifying ConnectivitySummary
3. Source and Configuration Management
The Git Version Control WorkflowInstallationSetting Up GitHub or GitLabCreating New DirectoriesInitializing LocallyCloning the Remote RepositoryUpdating README.mdCreating a New FileIgnoring FilesCreating a Development BranchAdding the BranchCreating a New FileSetting the Branch’s Remote OriginMerging the Branch into mainPerforming Optional CleanupRebasing and Resetting Your CodeCreating a New Test BranchModifying the Remote main BranchTemporarily Stashing ChangesCleanupCreating a Repository for the Book’s Project CodeSummary

Part II: Log Extraction and Management
4. Endpoint and Network Data
Collecting Logs with FilebeatInstallationEnabling TLSCreating a Configuration FileGenerating Certificate Signing RequestsConfigurationInput SourcesReading Local FilesApplying Parsers and New FieldsListening to the NetworkConnecting to External SystemsProcessorsControlling Processors with ConditionalsScreening Out DataModulesSending OutputsPublishing to KafkaPublishing to RedisWriting Output Data to a FileSending Data to LogstashPruning and Privatizing DataFilebeat for WindowsSummary
5. Windows Logs
Collecting Logs with WinlogbeatEvent Log TypesApplication and System LogsThe Security Log and SysmonOther Log SourcesInstallationEnabling TLSCreating a Configuration FileGenerating a Key Pair and Client CertificateConfigurationStarting the ServiceCollecting Logs from Sysmon and PowerShellSysmonPowerShell Script BlocksPowerShell ModulesExploring Available Event LogsWorking with ProcessorsSummary
6. Integrating and Storing Data
An Elastic Agent ArchitectureSetting Up the EnvironmentEnabling TLSConfiguring DNS and Firewall RulesInstalling Tools on Multiple ServersViewing Events in KibanaConfiguring Different Technology IntegrationsNetwork MetadataIptables Firewall LogsIntegration GuidelinesStand-Alone AgentsPreparing the Virtual MachineReceiving a Logstash API KeyConfiguring a Stand-Alone Agent PolicyConfiguring LogstashElasticsearch Ingest PipelinesCreating a Custom Ingest PipelineTesting the PipelineSummary
7. Working with Syslog Data
Logging PrioritiesCollecting Data with RsyslogInstallationEnabling TLSCustom Rsyslog ConfigurationsBasic vs. Advanced FormatsAdding and Testing New ConfigurationsAdvanced Format SyntaxGlobal PropertiesInput ModulesOutput ModulesTemplatesRulesetsExample ConfigurationsA ClientA Server RelaySummary
Part III: Data Transformation and Standardization
8. Data Manipulation Pipelines
Logstash Pipeline ComponentsLogstash Installation and SetupEnabling TLSConfiguring Java Virtual Machine OptionsInstalling CodecsInputs and OutputsHTTP POST RequestsAPI RequestsReading and Writing FilesKafkaRedisAmazon S3 and MinIOLogstash to LogstashNull OutputVirtual Addressing in PipelinesSummary
9. Transformation Filters
Logstash Filters for Data TransformationExtracting Data from Messy InputsExtracting Unstructured Data with grokExtracting Reliably Ordered Data with dissectHandling Inaccurate Syslog TimestampsEnriching DataNetwork Identification with the CIDR FilterKey-Value Lookups with translateAdding Directional TagsField ComparisonEquality and InequalityField ExistenceMembershipRegular ExpressionsParsing Key-Value Pairs from Structured Syslog DataSplitting Fields and Avoiding BackslashesInputMutateOutputWriting Filters with Ruby ScriptingInit and CodeArray ComparisonsDropping Data to Save CPUSummary
Part IV: Data Centralization, Automation, and Enrichment
10. Centralizing Security Data
Kafka FundamentalsProducers, Consumers, and Consumer GroupsBrokers and ControllersTopicsPartitionsData ReplicationListeners and MetadataStreams and ConnectorsCreating a Kafka ClusterSetting Up the NetworkCreating Users and DirectoriesInstalling Kafka and a Java Development KitEnabling TLSConfigurationsConfiguring Server PropertiesDefining the Connection SettingsCreating Cluster IDsUsing Systemd for Automatic StartupRunning KafkaStarting BrokersCreating TopicsPublishing MessagesSubscribing to TopicsCreating Tool-Specific TopicsConnecting to External ToolsRsyslogFilebeatLogstashKafka Pipeline Design ConsiderationsSummary
11. Automating Tool Configurations
Working with AnsibleInventoriesCommands, Playbooks, and RolesGathering System FactsTemplatesTemporary FilesSSH RequirementsInstallationCreating InventoriesConfiguration FilesRunning Ad Hoc CommandsEnabling Check ModeDistributing FilesExecuting CommandsUpdating PackagesManaging ServicesRebooting ServersPlaybooks for Task ExecutionReusable TasksDelegating TasksPerforming Tasks LocallyWriting Dynamic Data to FilesModifying hosts FilesAutomatically Generating Encryption KeysAppending TextSummary
12. Ansible Tasks and Playbooks
Allowing Firewall TrafficAutomatic File TransfersDistributing the Kafka PackageDownloading Files from the InternetCloning Git RepositoriesCreating Kafka Configuration TemplatesInstalling GPG KeysResponding to PromptsAccessing APIs and Web ServicesTLS AutomationSetting Up the DirectoryCreating the Root CACreating the Intermediate CAGenerating the CA ChainCreating a Flex CertificateAdding Container FilesCreating the TLS PlaybookTroubleshooting with debugSummary
13. Caching Threat Intelligence Data
Speeding Up Lookups with CachingInstalling Redis and MemcachedDividing the Project Servers into NodesEnabling TLSConfiguring RedisConfiguring MemcachedWorking with RedisWorking with MemcachedPopulating and Replicating Redis for Threat LookupsCreating the Leader CacheReceiving Threat Indicators with LogstashDefining InputsProcessing IndicatorsHandling Custom UploadsPopulating Redis with Key-Value PairsCreating Custom API KeysOutputting Data to ElasticsearchReading Threat IntelligenceCreating Redis FollowersConfiguring a Logstash GetterDeduplicating and Enriching DataDistributing Threat Intelligence with MemcachedConfiguring the Cyber Threat Intelligence NodeConfiguring the Data NodePreventing DriftHandling Analyst-Submitted Indicator DataGoing BeyondSummary
Index

Content preview from Data Engineering for Cybersecurity

7 WORKING WITH SYSLOG DATA

A robot pours a sack of jumbled pieces from various graphs into a funnel.

At the heart of Unix-like systems lies syslog, a message-logging format that has been a cornerstone of modern logging infrastructure since its introduction in the 1980s. Syslog messages contain important information about host actions, providing key insights for security analysts down the line. This chapter explores the format through Rsyslog, a high-performance, security-aware log collector used for working with syslog data.

We’ll begin by examining syslog’s logging priorities and consider how its severity levels and facilities work together to create a nuanced classification system for your log data. This information ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 0642572230111Errata Page

Cloud Computing