book

Financial Data Engineering

by Tamer Khraisha

October 2024

Intermediate to advanced

506 pages

15h 54m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This Book?PrerequisitesWhat to Expect from This BookBook Resources and ReferencesConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Defining Financial Data EngineeringFirst of All, What Is Finance?Defining Data EngineeringDefining Financial Data EngineeringWhy Financial Data Engineering?Volume, Variety, and Velocity of Financial DataFinance-Specific Data Requirements and ProblemsFinancial Machine LearningThe Disruptive FinTech LandscapeRegulatory Requirements and ComplianceThe Financial Data Engineer RoleDescription of the RoleWhere Do Financial Data Engineers Work?Responsibilities and Activities of a Financial Data EngineerSkills of a Financial Data EngineerSummary
Sources of Financial DataPublic Financial DataSecurity ExchangesCommercial Data Vendors, Providers, and DistributorsSurvey DataAlternative DataConfidential and Proprietary DataStructures of Financial DataTime Series DataCross-Sectional DataPanel DataMatrix DataGraph DataText DataTypes of Financial DataFundamental DataMarket DataTransaction DataAnalytics DataAlternative DataReference DataEntity DataBenchmark Financial DatasetsCenter for Research in Security PricesCompustat FinancialsTrade and Quote DatabaseInstitutional Brokers’ Estimate SystemIvyDB OptionMetricsTrade Reporting and Compliance EngineOrbis Global DatabaseSDC PlatinumStandard & Poor’s Dow Jones IndicesAlternative DatasetsSummary
Financial IdentifiersFinancial Identifier and Identification System DefinedThe Need for Financial IdentifiersWho Creates Financial Identification Systems?Desired Properties of a Financial IdentifierUniquenessGlobalityScalabilityCompletenessAccessibilityTimelinessAuthenticityGranularityPermanenceImmutabilitySecurityFinancial Identification Systems LandscapeInternational Securities Identification NumberClassification of Financial InstrumentsFinancial Instrument Short NameCommittee on Uniform Security Identification ProceduresLegal Entity IdentifierTransaction IdentifiersStock Exchange Daily Official ListTicker SymbolsDerivative IdentifiersFinancial Instrument Global IdentifierFactSet Permanent IdentifierLSEG Permanent IdentifierDigital Asset IdentifiersIndustry and Sector IdentifiersBank IdentifiersSummary
Financial Entity DefinedFinancial Named Entity RecognitionNamed Entity Recognition DescribedHow Does Named Entity Recognition Work?Approaches to Named Entity RecognitionNamed Entity Recognition Software LibrariesFinancial Entity ResolutionEntity Resolution DescribedThe Importance of Entity Resolution in FinanceHow Does Entity Resolution Work?Approaches to Entity ResolutionEntity Resolution Software LibrariesSummary
Financial Data GovernanceFinancial Data Governance DefinedFinancial Data Governance JustifiedData QualityDimension 1: Data ErrorsDimension 2: Data OutliersDimension 3: Data BiasesDimension 4: Data GranularityDimension 5: Data DuplicatesDimension 6: Data Availability and CompletenessDimension 7: Data TimelinessDimension 8: Data ConstraintsDimension 9: Data RelevanceData IntegrityPrinciple 1: Data StandardsPrinciple 2: Data BackupsPrinciple 3: Data ArchivingPrinciple 4: Data AggregationPrinciple 5: Data LineagePrinciple 6: Data CatalogsPrinciple 7: Data OwnershipPrinciple 8: Data ContractsPrinciple 9: Data ReconciliationData Security and PrivacyData PrivacyData AnonymizationData EncryptionAccess ControlSummary
Financial Data Engineering Lifecycle DefinedCriteria for Building the Financial Data Engineering StackCriterion 1: Open Source Versus Commercial SoftwareCriterion 2: Ease of Use Versus PerformanceCriterion 3: Cloud Versus On PremisesCriterion 4: Public Versus Private Versus Hybrid CloudCriterion 5: Single Versus Multi-CloudCriterion 6: Monolithic Versus Modular CodebaseSummary

Data Transmission and Arrival ProcessesData Transmission ProtocolsData Arrival ProcessesData Ingestion FormatsGeneral-Purpose FormatsBig Data FormatsIn-Memory FormatsStandardized Financial FormatsData Ingestion TechnologiesFinancial APIsFinancial Data FeedsSecure File TransferCloud AccessWeb AccessSpecialized Financial SoftwareData Ingestion Best PracticesMeet Business RequirementsDesign for ChangeEnforce Data GovernancePerform Benchmarking and Stress TestingSummary
Principles of Data Storage System DesignPrinciple 1: Business RequirementsPrinciple 2: Data ModelingPrinciple 3: Transactional GuaranteePrinciple 4: Consistency TradeoffsPrinciple 4: ScalabilityPrinciple 5: SecurityData Storage ModelingSQL Versus NoSQLPrimary Versus SecondaryOperational Versus AnalyticalNative Versus Non-NativeMulti-Model Versus Polyglot PersistenceData Storage ModelsThe Data Lake ModelThe Relational ModelThe Document ModelThe Time Series ModelThe Message Broker ModelThe Graph ModelThe Warehouse ModelThe Blockchain ModelSummary
Data Querying Querying PatternsQuery OptimizationData TransformationTransformation OperationsTransformation PatternsComputational RequirementsData DeliveryData ConsumersDelivery MechanismsSummary
Metrics, Events, Logs, and Traces MetricsEventsLogsTracesData Quality MonitoringPerformance MonitoringCost MonitoringBusiness and Analytical MonitoringData ObservabilitySummary
Workflow-Oriented Software ArchitecturesWhat Is a Data Workflow?Workflow Management SystemsFlexibility Configurability Dependency ManagementCoordination PatternsScalability IntegrationTypes of Financial Data WorkflowsExtract-Transform-Load WorkflowsStream Processing WorkflowsMicroservice WorkflowsMachine Learning WorkflowsSummary
PrerequisitesProject 1: Designing a Bank Account Management System Database with PostgreSQLConceptual Model: Business RequirementsLogical Model: Entity Relationship DiagramPhysical Model: Data Definition and Manipulation LanguageProject 1: Local TestingProject 1: Clean UpProject 1: SummaryProject 2: Designing a Financial Data ETL Workflow with Mage and PythonProject 2: Workflow DefinitionProject 2: Database DesignProject 2: Local TestingProject 2: Clean UpProject 2: SummaryProject 3: Designing a Microservice Workflow with Netflix Conductor, PostgreSQL, and PythonProject 3: Workflow DefinitionProject 3: Database DesignProject 3: Local TestingProject 3: Clean UpProject 3: SummaryProject 4: Designing a Financial Reference Data Store with OpenFIGI, PermID, and GLEIF APIsProject 4: PrerequisitesProject 4: Local TestingProject 4: Clean UpProject 4: SummaryConclusionFollow Updates on These ProjectsReport Issues or Ask Questions
Financial IntegrationDigitalization of Financial Markets and Cloud AdoptionFinancial RegulationFinancial Data Sharing and MarketplacesFinancial StandardizationArtificial Intelligence and Language ModelsArchitectures for Specific Business DomainsData CollectionSpeed and EfficiencyTokenization, Blockchain, and Digital CurrenciesWhat Can You Do Next?

Content preview from Financial Data Engineering

Chapter 11. Financial Data Workflows

Throughout this part of the book, you have explored the fundamental layers of the financial data engineering lifecycle: ingestion, storage, transformation and delivery, and monitoring. Adhering to this structured framework is crucial for building and maintaining efficient financial data infrastructures.

Importantly, as your company expands, the financial data infrastructure will become larger and more complex, generating many interdependencies and links between the different layers. To address this challenge, further refinement is required to build multiple independent and specialized data processing components. Data engineers call these components data pipelines or workflows.

This chapter will explore the foundational concepts of data workflows, introducing the ideas of workflow-oriented software architectures and workflow management systems. Then, it will cover the main types and characteristics of financial data workflows that can be implemented to handle financial data processing tasks.

Workflow-Oriented Software Architectures

Before explaining workflows, it’s important to understand the business rationale behind their practicality. First, as a natural part of the growth journey of any digital company, the technological stack tends to increase in size and complexity, creating numerous interdependencies, logical flows, and interactions among various system components. For this reason, the software development community has identified the ...