book

Financial Data Engineering

by Tamer Khraisha

October 2024

Intermediate to advanced

506 pages

15h 54m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This Book?PrerequisitesWhat to Expect from This BookBook Resources and ReferencesConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Defining Financial Data EngineeringFirst of All, What Is Finance?Defining Data EngineeringDefining Financial Data EngineeringWhy Financial Data Engineering?Volume, Variety, and Velocity of Financial DataFinance-Specific Data Requirements and ProblemsFinancial Machine LearningThe Disruptive FinTech LandscapeRegulatory Requirements and ComplianceThe Financial Data Engineer RoleDescription of the RoleWhere Do Financial Data Engineers Work?Responsibilities and Activities of a Financial Data EngineerSkills of a Financial Data EngineerSummary
Sources of Financial DataPublic Financial DataSecurity ExchangesCommercial Data Vendors, Providers, and DistributorsSurvey DataAlternative DataConfidential and Proprietary DataStructures of Financial DataTime Series DataCross-Sectional DataPanel DataMatrix DataGraph DataText DataTypes of Financial DataFundamental DataMarket DataTransaction DataAnalytics DataAlternative DataReference DataEntity DataBenchmark Financial DatasetsCenter for Research in Security PricesCompustat FinancialsTrade and Quote DatabaseInstitutional Brokers’ Estimate SystemIvyDB OptionMetricsTrade Reporting and Compliance EngineOrbis Global DatabaseSDC PlatinumStandard & Poor’s Dow Jones IndicesAlternative DatasetsSummary
Financial IdentifiersFinancial Identifier and Identification System DefinedThe Need for Financial IdentifiersWho Creates Financial Identification Systems?Desired Properties of a Financial IdentifierUniquenessGlobalityScalabilityCompletenessAccessibilityTimelinessAuthenticityGranularityPermanenceImmutabilitySecurityFinancial Identification Systems LandscapeInternational Securities Identification NumberClassification of Financial InstrumentsFinancial Instrument Short NameCommittee on Uniform Security Identification ProceduresLegal Entity IdentifierTransaction IdentifiersStock Exchange Daily Official ListTicker SymbolsDerivative IdentifiersFinancial Instrument Global IdentifierFactSet Permanent IdentifierLSEG Permanent IdentifierDigital Asset IdentifiersIndustry and Sector IdentifiersBank IdentifiersSummary
Financial Entity DefinedFinancial Named Entity RecognitionNamed Entity Recognition DescribedHow Does Named Entity Recognition Work?Approaches to Named Entity RecognitionNamed Entity Recognition Software LibrariesFinancial Entity ResolutionEntity Resolution DescribedThe Importance of Entity Resolution in FinanceHow Does Entity Resolution Work?Approaches to Entity ResolutionEntity Resolution Software LibrariesSummary
Financial Data GovernanceFinancial Data Governance DefinedFinancial Data Governance JustifiedData QualityDimension 1: Data ErrorsDimension 2: Data OutliersDimension 3: Data BiasesDimension 4: Data GranularityDimension 5: Data DuplicatesDimension 6: Data Availability and CompletenessDimension 7: Data TimelinessDimension 8: Data ConstraintsDimension 9: Data RelevanceData IntegrityPrinciple 1: Data StandardsPrinciple 2: Data BackupsPrinciple 3: Data ArchivingPrinciple 4: Data AggregationPrinciple 5: Data LineagePrinciple 6: Data CatalogsPrinciple 7: Data OwnershipPrinciple 8: Data ContractsPrinciple 9: Data ReconciliationData Security and PrivacyData PrivacyData AnonymizationData EncryptionAccess ControlSummary
Financial Data Engineering Lifecycle DefinedCriteria for Building the Financial Data Engineering StackCriterion 1: Open Source Versus Commercial SoftwareCriterion 2: Ease of Use Versus PerformanceCriterion 3: Cloud Versus On PremisesCriterion 4: Public Versus Private Versus Hybrid CloudCriterion 5: Single Versus Multi-CloudCriterion 6: Monolithic Versus Modular CodebaseSummary

Data Transmission and Arrival ProcessesData Transmission ProtocolsData Arrival ProcessesData Ingestion FormatsGeneral-Purpose FormatsBig Data FormatsIn-Memory FormatsStandardized Financial FormatsData Ingestion TechnologiesFinancial APIsFinancial Data FeedsSecure File TransferCloud AccessWeb AccessSpecialized Financial SoftwareData Ingestion Best PracticesMeet Business RequirementsDesign for ChangeEnforce Data GovernancePerform Benchmarking and Stress TestingSummary
Principles of Data Storage System DesignPrinciple 1: Business RequirementsPrinciple 2: Data ModelingPrinciple 3: Transactional GuaranteePrinciple 4: Consistency TradeoffsPrinciple 4: ScalabilityPrinciple 5: SecurityData Storage ModelingSQL Versus NoSQLPrimary Versus SecondaryOperational Versus AnalyticalNative Versus Non-NativeMulti-Model Versus Polyglot PersistenceData Storage ModelsThe Data Lake ModelThe Relational ModelThe Document ModelThe Time Series ModelThe Message Broker ModelThe Graph ModelThe Warehouse ModelThe Blockchain ModelSummary
Data Querying Querying PatternsQuery OptimizationData TransformationTransformation OperationsTransformation PatternsComputational RequirementsData DeliveryData ConsumersDelivery MechanismsSummary
Metrics, Events, Logs, and Traces MetricsEventsLogsTracesData Quality MonitoringPerformance MonitoringCost MonitoringBusiness and Analytical MonitoringData ObservabilitySummary
Workflow-Oriented Software ArchitecturesWhat Is a Data Workflow?Workflow Management SystemsFlexibility Configurability Dependency ManagementCoordination PatternsScalability IntegrationTypes of Financial Data WorkflowsExtract-Transform-Load WorkflowsStream Processing WorkflowsMicroservice WorkflowsMachine Learning WorkflowsSummary
PrerequisitesProject 1: Designing a Bank Account Management System Database with PostgreSQLConceptual Model: Business RequirementsLogical Model: Entity Relationship DiagramPhysical Model: Data Definition and Manipulation LanguageProject 1: Local TestingProject 1: Clean UpProject 1: SummaryProject 2: Designing a Financial Data ETL Workflow with Mage and PythonProject 2: Workflow DefinitionProject 2: Database DesignProject 2: Local TestingProject 2: Clean UpProject 2: SummaryProject 3: Designing a Microservice Workflow with Netflix Conductor, PostgreSQL, and PythonProject 3: Workflow DefinitionProject 3: Database DesignProject 3: Local TestingProject 3: Clean UpProject 3: SummaryProject 4: Designing a Financial Reference Data Store with OpenFIGI, PermID, and GLEIF APIsProject 4: PrerequisitesProject 4: Local TestingProject 4: Clean UpProject 4: SummaryConclusionFollow Updates on These ProjectsReport Issues or Ask Questions
Financial IntegrationDigitalization of Financial Markets and Cloud AdoptionFinancial RegulationFinancial Data Sharing and MarketplacesFinancial StandardizationArtificial Intelligence and Language ModelsArchitectures for Specific Business DomainsData CollectionSpeed and EfficiencyTokenization, Blockchain, and Digital CurrenciesWhat Can You Do Next?

Content preview from Financial Data Engineering

Chapter 12. Hands-On Projects

Now that you’ve gained a fundamental grasp of financial data engineering, it’s time to put it into practice. In this chapter, you will go through a series of practical projects designed to give you firsthand experience working with financial data.

Four projects will be discussed, each focusing on a different problem and employing a unique technological stack:

Constructing a bank account management system with PostgreSQL
Building a financial data ETL workflow with Mage
Developing a financial microservice workflow with Netflix Conductor
Implementing a reference data store with OpenFIGI, PermID, and GLEIF APIs

A few points should be mentioned about these projects. First, they are meant to provide hands-on experience with financial data engineering and may not necessarily represent complete solutions for real business problems. Additionally, they are intended to be executed locally on your machine and not deployed in a production environment. Finally, the employed technologies are not indicative of the author’s personal preferences; instead, they reflect the author’s prudent consideration of what would best clarify the problems and solutions for the reader.

Prerequisites

All projects in this chapter will be packaged and isolated with all their dependencies into Docker containers. Docker is the most popular open source software for operating system (OS) virtualization (aka containerization). It is available for Windows, macOS, and Linux through ...