book

Financial Data Engineering

by Tamer Khraisha

October 2024

Intermediate to advanced

506 pages

15h 54m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This Book?PrerequisitesWhat to Expect from This BookBook Resources and ReferencesConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Defining Financial Data EngineeringFirst of All, What Is Finance?Defining Data EngineeringDefining Financial Data EngineeringWhy Financial Data Engineering?Volume, Variety, and Velocity of Financial DataFinance-Specific Data Requirements and ProblemsFinancial Machine LearningThe Disruptive FinTech LandscapeRegulatory Requirements and ComplianceThe Financial Data Engineer RoleDescription of the RoleWhere Do Financial Data Engineers Work?Responsibilities and Activities of a Financial Data EngineerSkills of a Financial Data EngineerSummary
Sources of Financial DataPublic Financial DataSecurity ExchangesCommercial Data Vendors, Providers, and DistributorsSurvey DataAlternative DataConfidential and Proprietary DataStructures of Financial DataTime Series DataCross-Sectional DataPanel DataMatrix DataGraph DataText DataTypes of Financial DataFundamental DataMarket DataTransaction DataAnalytics DataAlternative DataReference DataEntity DataBenchmark Financial DatasetsCenter for Research in Security PricesCompustat FinancialsTrade and Quote DatabaseInstitutional Brokers’ Estimate SystemIvyDB OptionMetricsTrade Reporting and Compliance EngineOrbis Global DatabaseSDC PlatinumStandard & Poor’s Dow Jones IndicesAlternative DatasetsSummary
Financial IdentifiersFinancial Identifier and Identification System DefinedThe Need for Financial IdentifiersWho Creates Financial Identification Systems?Desired Properties of a Financial IdentifierUniquenessGlobalityScalabilityCompletenessAccessibilityTimelinessAuthenticityGranularityPermanenceImmutabilitySecurityFinancial Identification Systems LandscapeInternational Securities Identification NumberClassification of Financial InstrumentsFinancial Instrument Short NameCommittee on Uniform Security Identification ProceduresLegal Entity IdentifierTransaction IdentifiersStock Exchange Daily Official ListTicker SymbolsDerivative IdentifiersFinancial Instrument Global IdentifierFactSet Permanent IdentifierLSEG Permanent IdentifierDigital Asset IdentifiersIndustry and Sector IdentifiersBank IdentifiersSummary
Financial Entity DefinedFinancial Named Entity RecognitionNamed Entity Recognition DescribedHow Does Named Entity Recognition Work?Approaches to Named Entity RecognitionNamed Entity Recognition Software LibrariesFinancial Entity ResolutionEntity Resolution DescribedThe Importance of Entity Resolution in FinanceHow Does Entity Resolution Work?Approaches to Entity ResolutionEntity Resolution Software LibrariesSummary
Financial Data GovernanceFinancial Data Governance DefinedFinancial Data Governance JustifiedData QualityDimension 1: Data ErrorsDimension 2: Data OutliersDimension 3: Data BiasesDimension 4: Data GranularityDimension 5: Data DuplicatesDimension 6: Data Availability and CompletenessDimension 7: Data TimelinessDimension 8: Data ConstraintsDimension 9: Data RelevanceData IntegrityPrinciple 1: Data StandardsPrinciple 2: Data BackupsPrinciple 3: Data ArchivingPrinciple 4: Data AggregationPrinciple 5: Data LineagePrinciple 6: Data CatalogsPrinciple 7: Data OwnershipPrinciple 8: Data ContractsPrinciple 9: Data ReconciliationData Security and PrivacyData PrivacyData AnonymizationData EncryptionAccess ControlSummary
Financial Data Engineering Lifecycle DefinedCriteria for Building the Financial Data Engineering StackCriterion 1: Open Source Versus Commercial SoftwareCriterion 2: Ease of Use Versus PerformanceCriterion 3: Cloud Versus On PremisesCriterion 4: Public Versus Private Versus Hybrid CloudCriterion 5: Single Versus Multi-CloudCriterion 6: Monolithic Versus Modular CodebaseSummary

Data Transmission and Arrival ProcessesData Transmission ProtocolsData Arrival ProcessesData Ingestion FormatsGeneral-Purpose FormatsBig Data FormatsIn-Memory FormatsStandardized Financial FormatsData Ingestion TechnologiesFinancial APIsFinancial Data FeedsSecure File TransferCloud AccessWeb AccessSpecialized Financial SoftwareData Ingestion Best PracticesMeet Business RequirementsDesign for ChangeEnforce Data GovernancePerform Benchmarking and Stress TestingSummary
Principles of Data Storage System DesignPrinciple 1: Business RequirementsPrinciple 2: Data ModelingPrinciple 3: Transactional GuaranteePrinciple 4: Consistency TradeoffsPrinciple 4: ScalabilityPrinciple 5: SecurityData Storage ModelingSQL Versus NoSQLPrimary Versus SecondaryOperational Versus AnalyticalNative Versus Non-NativeMulti-Model Versus Polyglot PersistenceData Storage ModelsThe Data Lake ModelThe Relational ModelThe Document ModelThe Time Series ModelThe Message Broker ModelThe Graph ModelThe Warehouse ModelThe Blockchain ModelSummary
Data Querying Querying PatternsQuery OptimizationData TransformationTransformation OperationsTransformation PatternsComputational RequirementsData DeliveryData ConsumersDelivery MechanismsSummary
Metrics, Events, Logs, and Traces MetricsEventsLogsTracesData Quality MonitoringPerformance MonitoringCost MonitoringBusiness and Analytical MonitoringData ObservabilitySummary
Workflow-Oriented Software ArchitecturesWhat Is a Data Workflow?Workflow Management SystemsFlexibility Configurability Dependency ManagementCoordination PatternsScalability IntegrationTypes of Financial Data WorkflowsExtract-Transform-Load WorkflowsStream Processing WorkflowsMicroservice WorkflowsMachine Learning WorkflowsSummary
PrerequisitesProject 1: Designing a Bank Account Management System Database with PostgreSQLConceptual Model: Business RequirementsLogical Model: Entity Relationship DiagramPhysical Model: Data Definition and Manipulation LanguageProject 1: Local TestingProject 1: Clean UpProject 1: SummaryProject 2: Designing a Financial Data ETL Workflow with Mage and PythonProject 2: Workflow DefinitionProject 2: Database DesignProject 2: Local TestingProject 2: Clean UpProject 2: SummaryProject 3: Designing a Microservice Workflow with Netflix Conductor, PostgreSQL, and PythonProject 3: Workflow DefinitionProject 3: Database DesignProject 3: Local TestingProject 3: Clean UpProject 3: SummaryProject 4: Designing a Financial Reference Data Store with OpenFIGI, PermID, and GLEIF APIsProject 4: PrerequisitesProject 4: Local TestingProject 4: Clean UpProject 4: SummaryConclusionFollow Updates on These ProjectsReport Issues or Ask Questions
Financial IntegrationDigitalization of Financial Markets and Cloud AdoptionFinancial RegulationFinancial Data Sharing and MarketplacesFinancial StandardizationArtificial Intelligence and Language ModelsArchitectures for Specific Business DomainsData CollectionSpeed and EfficiencyTokenization, Blockchain, and Digital CurrenciesWhat Can You Do Next?

Overview

Today, investment in financial technology and digital transformation is reshaping the financial landscape and generating many opportunities. Too often, however, engineers and professionals in financial institutions lack a practical and comprehensive understanding of the concepts, problems, techniques, and technologies necessary to build a modern, reliable, and scalable financial data infrastructure. This is where financial data engineering is needed.

A data engineer developing a data infrastructure for a financial product possesses not only technical data engineering skills but also a solid understanding of financial domain-specific challenges, methodologies, data ecosystems, providers, formats, technological constraints, identifiers, entities, standards, regulatory requirements, and governance.

This book offers a comprehensive, practical, domain-driven approach to financial data engineering, featuring real-world use cases, industry practices, and hands-on projects.

You'll learn:

The data engineering landscape in the financial sector

Specific problems encountered in financial data engineering

The structure, players, and particularities of the financial data domain

Approaches to designing financial data identification and entity systems

Financial data governance frameworks, concepts, and best practices

The financial data engineering lifecycle from ingestion to production

The varieties and main characteristics of financial data workflows

How to build financial data pipelines using open source tools and APIs

Tamer Khraisha, PhD, is a senior data engineer and scientific author with more than a decade of experience in the financial sector.