book

Data Management at Scale

by Piethein Strengholt

July 2020

Intermediate to advanced

345 pages

10h 47m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Who Is This Book For?What Will I Learn?Navigating Through This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Data ManagementAnalytics Is Fragmenting the Data LandscapeSpeed of Software Delivery Is ChangingNetworks Are Getting FasterPrivacy and Security Concerns Are a Top PriorityOperational and Transactional Systems Need to Be IntegratedData Monetization Requires an Ecosystem-to-Ecosystem ArchitectureEnterprises Are Saddled with Outdated Data ArchitecturesEnterprise Data Warehouse and Business IntelligenceData LakeCentralized ViewSummary
Universally Acknowledged Starting PointsEach Application Has an Application DatabaseApplications Are Specific and Have Unique ContextGolden SourceThere’s No Escape from the Data Integration DilemmaApplications Play the Roles of Data Providers and Data ConsumersKey Theoretical ConsiderationsObject-Oriented Programming PrinciplesDomain-Driven DesignBusiness ArchitectureCommunication and Integration PatternsPoint-to-PointSilosHub-Spoke ModelScaled ArchitectureGolden Sources and Domain Data StoresData Delivery Contracts and Data Sharing AgreementsEliminating the Siloed ApproachDomain-Driven Design on an Enterprise ScaleRead-Optimized DataData Layer as a Holistic PictureMetadata and the Target Operating ModelSummary
Introducing the RDS ArchitectureCommand and Query Responsibility SegregationWhat Is CQRS?CQRS at ScaleRead-Only Data Store Components and ServicesMetadataData QualityRDS TiersData IngestionIntegrating Commercial Off-the-Shelf SolutionsExtracting Data from External APIs and SaaSsHistorical Data ServiceDesign VariationsData ReplicationAccess LayerFile Manipulation ServiceDelivery Notification ServiceDe-Identification ServiceDistributed OrchestrationIntelligent Consumption ServicesPopulating RDSs on DemandRDS Direct Usage ConsiderationsSummary
Introducing the API ArchitectureWhat Is Service-Oriented Architecture?Enterprise Application IntegrationService OrchestrationService ChoreographyPublic Services and Private ServicesService Models and Canonical Data ModelsSimilarities Between SOA and Enterprise Data Warehousing ArchitectureModern View on SOAAPI GatewayResponsibility ModelThe New Role of the ESBService ContractsService DiscoveryMicroservicesThe Role of the API Gateway Within MicroservicesFunctionsService MeshMicroservices BoundariesMicroservices Within the API Reference ArchitectureEcosystem CommunicationAPI-Based Communication ChannelsGraphQLBackend for FrontendMetadataUsing RDSs for Real-Time and Intensive ReadsSummary
Introducing the Streaming ArchitectureThe Asynchronous Event Model Makes the DifferenceWhat Do Event-Driven Architectures Look Like?Mediator TopologyBroker TopologyEvent Processing StylesA Gentle Introduction to Apache KafkaDistributed Event DataApache Kafka FeaturesThe Streaming ArchitectureEvent ProducersEvent ConsumersEvent PlatformEvent Sourcing and Command SourcingGovernance ModelBusiness StreamsStreaming Consumption PatternsEvent-Carried State TransferPlaying the Role of an RDSUsing Streaming to Populate RDSsControls and Policies for Guiding the DomainsStreaming as the Operational BackboneGuarantees and ConsistencyConsistency Level“At Least Once, Exactly Once, and at Most Once” ProcessingMessage OrderDead Letter QueueStreaming InteroperabilityMetadata for Governance and Self-Service ModelsSummary
Recap of the ArchitecturesRDS ArchitectureAPI ArchitectureStreaming ArchitectureStrengthening PatternsEnterprise Interoperability StandardsStable Data EndpointsData Delivery ContractsAccessible and Addressable DataCrossing Network PrinciplesEnterprise Data StandardsConsumption-Optimization PrinciplesDiscoverability of MetadataSemantic ConsistencySupplying the Corresponding MetadataData Origination and MovementsReference ArchitectureSummary
Data GovernanceOrganization: Data Governance RolesProcesses: Data Governance ActivitiesPeople: Trust and Ethical, Social, and Economic ConsiderationsTechnology: Golden Source, Ownership, and Application AdministrationData: Golden Sources, Golden Datasets, and ClassificationsData SecurityCurrent Siloed ApproachUnified Data Security for ArchitecturesIdentity ProvidersSecurity Reference Architecture and Data Context ApproachSecurity Process FlowPractical GuidanceRDS ArchitectureAPI ArchitectureStreaming ArchitectureIntelligent Learning EngineSummary
Consumption PatternsUsing Read-Only Data Stores DirectlyDomain Data StoresTarget Operating ModelData Professionals as a Target User GroupBusiness RequirementsNonfunctional RequirementsBuilding the Data Pipeline and Data ModelDistributing Integrated DataBusiness Intelligence CapabilitiesSelf-Service CapabilitiesAnalytical CapabilitiesStandard Infrastructure for Automated DeploymentsStateless ModelsPrescripted and Configured WorkbenchesStandardize on Model Integration PatternsAutomationModel MetadataAdvanced Analytics Reference ArchitectureSummary

Demystifying Master Data ManagementMaster Data Management StylesMDM Reference ArchitectureDesigning a Master Data Management SolutionMDM DistributionMaster Identification NumbersReference Data Versus Master DataDetermining the Scope of Your Enterprise DataMDM and Data Quality as a ServiceCurated DataMetadata ExchangeIntegrated ViewsReusable Components and Integration LogicData RepublishingRelation to Data GovernanceSummary
Metadata ManagementEnterprise Metadata ModelEnterprise Knowledge GraphArchitectural Approaches for Metadata ManagementMetadata InteroperabilityMetadata RepositoriesMarketplace to Provide Rapid Access to Authorized DataSummary
Delivery ModelFully Decentralized ApproachPartially Decentralized ApproachStructuring TeamsInnerSource StrategyCultureTechnology ChoicesThe Decline of Traditional Enterprise ArchitectureBlueprints and DiagramsModern SkillsControl and GovernanceLast Words

Content preview from Data Management at Scale

Preface

Social media, live streaming, and smartphones are just a few of the many ways digitization has changed all of our lives in recent years—and the pace is still accelerating, with digital transformations in music and television, shopping, and travel, among many other industries, while advancements in artificial intelligence and machine learning drive the growth of autonomous machines such as drones and self-driving cars.

What fuels this digital society? Data. During the 20th century, oil was the world’s most valuable resource. Today, data is the new oil. It’s only a matter of time until the growth of analytics pushes demand for data to levels we haven’t seen before.

As the amount of data generated skyrockets, so does its complexity. Trends like cloud, API management, microservices, open data, software-as-a-service (SaaS), and new software delivery models are on the rise, and countless new databases and analytical applications have been released over the last few years.

The sheer number of new approaches to data is fragmenting the digital landscape. We see more point-to-point interfaces, endless discussions about data quality and ownership, and plenty of ethical and legal dilemmas regarding privacy, safety, and security. Agility, long-term stability, and clear data governance compete with the need to develop new business cases swiftly. Our industry sorely needs a clear vision for the future of data management and integration.

This book’s perspective on data management and ...