book

Financial Data Engineering

by Tamer Khraisha

October 2024

Intermediate to advanced

506 pages

15h 54m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Who Should Read This Book?PrerequisitesWhat to Expect from This BookBook Resources and ReferencesConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Defining Financial Data EngineeringFirst of All, What Is Finance?Defining Data EngineeringDefining Financial Data EngineeringWhy Financial Data Engineering?Volume, Variety, and Velocity of Financial DataFinance-Specific Data Requirements and ProblemsFinancial Machine LearningThe Disruptive FinTech LandscapeRegulatory Requirements and ComplianceThe Financial Data Engineer RoleDescription of the RoleWhere Do Financial Data Engineers Work?Responsibilities and Activities of a Financial Data EngineerSkills of a Financial Data EngineerSummary
Sources of Financial DataPublic Financial DataSecurity ExchangesCommercial Data Vendors, Providers, and DistributorsSurvey DataAlternative DataConfidential and Proprietary DataStructures of Financial DataTime Series DataCross-Sectional DataPanel DataMatrix DataGraph DataText DataTypes of Financial DataFundamental DataMarket DataTransaction DataAnalytics DataAlternative DataReference DataEntity DataBenchmark Financial DatasetsCenter for Research in Security PricesCompustat FinancialsTrade and Quote DatabaseInstitutional Brokers’ Estimate SystemIvyDB OptionMetricsTrade Reporting and Compliance EngineOrbis Global DatabaseSDC PlatinumStandard & Poor’s Dow Jones IndicesAlternative DatasetsSummary
Financial IdentifiersFinancial Identifier and Identification System DefinedThe Need for Financial IdentifiersWho Creates Financial Identification Systems?Desired Properties of a Financial IdentifierUniquenessGlobalityScalabilityCompletenessAccessibilityTimelinessAuthenticityGranularityPermanenceImmutabilitySecurityFinancial Identification Systems LandscapeInternational Securities Identification NumberClassification of Financial InstrumentsFinancial Instrument Short NameCommittee on Uniform Security Identification ProceduresLegal Entity IdentifierTransaction IdentifiersStock Exchange Daily Official ListTicker SymbolsDerivative IdentifiersFinancial Instrument Global IdentifierFactSet Permanent IdentifierLSEG Permanent IdentifierDigital Asset IdentifiersIndustry and Sector IdentifiersBank IdentifiersSummary
Financial Entity DefinedFinancial Named Entity RecognitionNamed Entity Recognition DescribedHow Does Named Entity Recognition Work?Approaches to Named Entity RecognitionNamed Entity Recognition Software LibrariesFinancial Entity ResolutionEntity Resolution DescribedThe Importance of Entity Resolution in FinanceHow Does Entity Resolution Work?Approaches to Entity ResolutionEntity Resolution Software LibrariesSummary
Financial Data GovernanceFinancial Data Governance DefinedFinancial Data Governance JustifiedData QualityDimension 1: Data ErrorsDimension 2: Data OutliersDimension 3: Data BiasesDimension 4: Data GranularityDimension 5: Data DuplicatesDimension 6: Data Availability and CompletenessDimension 7: Data TimelinessDimension 8: Data ConstraintsDimension 9: Data RelevanceData IntegrityPrinciple 1: Data StandardsPrinciple 2: Data BackupsPrinciple 3: Data ArchivingPrinciple 4: Data AggregationPrinciple 5: Data LineagePrinciple 6: Data CatalogsPrinciple 7: Data OwnershipPrinciple 8: Data ContractsPrinciple 9: Data ReconciliationData Security and PrivacyData PrivacyData AnonymizationData EncryptionAccess ControlSummary
Financial Data Engineering Lifecycle DefinedCriteria for Building the Financial Data Engineering StackCriterion 1: Open Source Versus Commercial SoftwareCriterion 2: Ease of Use Versus PerformanceCriterion 3: Cloud Versus On PremisesCriterion 4: Public Versus Private Versus Hybrid CloudCriterion 5: Single Versus Multi-CloudCriterion 6: Monolithic Versus Modular CodebaseSummary

Data Transmission and Arrival ProcessesData Transmission ProtocolsData Arrival ProcessesData Ingestion FormatsGeneral-Purpose FormatsBig Data FormatsIn-Memory FormatsStandardized Financial FormatsData Ingestion TechnologiesFinancial APIsFinancial Data FeedsSecure File TransferCloud AccessWeb AccessSpecialized Financial SoftwareData Ingestion Best PracticesMeet Business RequirementsDesign for ChangeEnforce Data GovernancePerform Benchmarking and Stress TestingSummary
Principles of Data Storage System DesignPrinciple 1: Business RequirementsPrinciple 2: Data ModelingPrinciple 3: Transactional GuaranteePrinciple 4: Consistency TradeoffsPrinciple 4: ScalabilityPrinciple 5: SecurityData Storage ModelingSQL Versus NoSQLPrimary Versus SecondaryOperational Versus AnalyticalNative Versus Non-NativeMulti-Model Versus Polyglot PersistenceData Storage ModelsThe Data Lake ModelThe Relational ModelThe Document ModelThe Time Series ModelThe Message Broker ModelThe Graph ModelThe Warehouse ModelThe Blockchain ModelSummary
Data Querying Querying PatternsQuery OptimizationData TransformationTransformation OperationsTransformation PatternsComputational RequirementsData DeliveryData ConsumersDelivery MechanismsSummary
Metrics, Events, Logs, and Traces MetricsEventsLogsTracesData Quality MonitoringPerformance MonitoringCost MonitoringBusiness and Analytical MonitoringData ObservabilitySummary
Workflow-Oriented Software ArchitecturesWhat Is a Data Workflow?Workflow Management SystemsFlexibility Configurability Dependency ManagementCoordination PatternsScalability IntegrationTypes of Financial Data WorkflowsExtract-Transform-Load WorkflowsStream Processing WorkflowsMicroservice WorkflowsMachine Learning WorkflowsSummary
PrerequisitesProject 1: Designing a Bank Account Management System Database with PostgreSQLConceptual Model: Business RequirementsLogical Model: Entity Relationship DiagramPhysical Model: Data Definition and Manipulation LanguageProject 1: Local TestingProject 1: Clean UpProject 1: SummaryProject 2: Designing a Financial Data ETL Workflow with Mage and PythonProject 2: Workflow DefinitionProject 2: Database DesignProject 2: Local TestingProject 2: Clean UpProject 2: SummaryProject 3: Designing a Microservice Workflow with Netflix Conductor, PostgreSQL, and PythonProject 3: Workflow DefinitionProject 3: Database DesignProject 3: Local TestingProject 3: Clean UpProject 3: SummaryProject 4: Designing a Financial Reference Data Store with OpenFIGI, PermID, and GLEIF APIsProject 4: PrerequisitesProject 4: Local TestingProject 4: Clean UpProject 4: SummaryConclusionFollow Updates on These ProjectsReport Issues or Ask Questions
Financial IntegrationDigitalization of Financial Markets and Cloud AdoptionFinancial RegulationFinancial Data Sharing and MarketplacesFinancial StandardizationArtificial Intelligence and Language ModelsArchitectures for Specific Business DomainsData CollectionSpeed and EfficiencyTokenization, Blockchain, and Digital CurrenciesWhat Can You Do Next?

Content preview from Financial Data Engineering

Part II. The Financial Data Engineering Lifecycle

In the first part of this book, we explored the major ideas and problems around the management of financial data, including the complexity of the financial data landscape, the diversity and nonuniversality of financial data identifiers, the problems of financial entity recognition and resolution, and financial data governance.

In the second part of this book, spanning Chapters 6 through 12, the focus shifts to the technological aspects of financial data engineering. This includes the models, tools, frameworks, software systems, hardware components, libraries, design patterns, and networking systems needed to design and implement a financial data infrastructure.

Chapter 6 will begin by introducing the financial data engineering lifecycle (FDEL), a conceptual framework that will be used to organize the many components of a financial data infrastructure into four structured layers: ingestion, storage, transformation and delivery, and monitoring. Subsequently, Chapters 7 through 10 will cover each of these layers in detail. After that, Chapter 11 will discuss the various workflow architectures that are commonly used to implement the FDEL. Chapter 12 concludes with four hands-on projects designed to familiarize you with key practices and technologies in financial data engineering.