book

Data Management at Scale, 2nd Edition

by Piethein Strengholt

April 2023

Beginner to intermediate

412 pages

12h 48m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Why I Wrote This Book and Why NowWho Is This Book For?How to Read or Use This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Recent Technology Developments and Industry TrendsData ManagementAnalytics Is Fragmenting the Data LandscapeThe Speed of Software Delivery Is ChangingThe Cloud’s Impact on Data Management Is ImmeasurablePrivacy and Security Concerns Are a Top PriorityOperational and Analytical Systems Need to Be IntegratedOrganizations Operate in Collaborative EcosystemsEnterprises Are Saddled with Outdated Data ArchitecturesThe Enterprise Data Warehouse: A Single Source of TruthThe Data Lake: A Centralized Repository for Structured and Unstructured DataThe Pain of CentralizationDefining a Data StrategyWrapping Up
Application Design Starting PointsEach Application Has a Data StoreApplications Are Always UniqueGolden SourcesThe Data Integration DilemmaApplication RolesInspirations from Software ArchitectureData DomainsDomain-Driven DesignBusiness ArchitectureDomain CharacteristicsPrinciples for Distributed and Domain-Oriented Data ManagementDesign Principles for Data DomainsBest Practices for Data ProvidersDomain Ownership ResponsibilitiesTransitioning Toward Distributed and Domain-Oriented Data ManagementWrapping Up
Domain Topologies: Managing Problem SpacesFully Federated Domain TopologyGoverned Domain TopologyPartially Federated Domain TopologyValue Chain–Aligned Domain TopologyCoarse-Grained Domain TopologyCoarse-Grained and Partially Governed Domain TopologyCentralized Domain TopologyPicking the Right TopologyLanding Zone Topologies: Managing Solution SpacesSingle Data Landing ZoneSource- and Consumer-Aligned Landing ZonesHub Data Landing ZoneMultiple Data Landing ZonesMultiple Data Management Landing ZonesPractical Landing Zones ExampleWrapping Up
What Are Data Products?Problems with Combining Code, Data, Metadata, and InfrastructureData Products as Logical EntitiesData Product Design PatternsWhat Is CQRS?Read Replicas as Data ProductsDesign Principles for Data ProductsResource-Oriented Read-Optimized DesignData Product Data Is ImmutableUsing the Ubiquitous LanguageCapture Directly from the SourceClear Interoperability StandardsNo Raw DataDon’t Conform to ConsumersMissing Values, Defaults, and Data TypesSemantic ConsistencyAtomicityCompatibilityAbstract Volatile Reference DataNew Data Means New OwnershipData Security PatternsEstablish a MetamodelAllow Self-ServiceCross-Domain RelationshipsEnterprise ConsistencyHistorization, Redeliveries, and OverwritesBusiness Capabilities with Multiple OwnersOperating ModelData Product ArchitectureHigh-Level Platform DesignCapabilities for Capturing and Onboarding DataData QualityData HistorizationSolution DesignReal-World ExampleAlignment with Storage AccountsAlignment with Data PipelinesCapabilities for Serving DataData Serving ServicesFile Manipulation ServiceDe-Identification ServiceDistributed OrchestrationIntelligent Consumption ServicesDirect Usage ConsiderationsGetting StartedWrapping Up
Introducing API ManagementWhat Is Service-Oriented Architecture?Enterprise Application IntegrationService OrchestrationService ChoreographyPublic Services and Private ServicesService Models and Canonical Data ModelsParallels with Enterprise Data Warehousing ArchitectureA Modern View of API ManagementFederated Responsibility ModelAPI GatewayAPI as a ProductComposite ServicesAPI ContractsAPI DiscoverabilityMicroservicesFunctionsService MeshMicroservice Domain BoundariesEcosystem CommunicationExperience APIsGraphQLBackend for FrontendPractical ExampleMetadata ManagementRead-Oriented APIs Serving Data ProductsWrapping Up
Introduction to EventsNotifications Versus Carried StateThe Asynchronous Communication ModelWhat Do Modern Event-Driven Architectures Look Like?Message QueuesEvent BrokersEvent Processing StylesEvent ProducersEvent ConsumersEvent Streaming PlatformsGovernance ModelEvent Stores as Data Product StoresEvent Stores as Application BackendsStreaming as the Operational BackboneGuarantees and ConsistencyConsistency LevelProcessing MethodsMessage OrderDead Letter QueueStreaming InteroperabilityGovernance and Self-ServiceWrapping Up
Cross-Domain InteroperabilityQuick RecapData Distribution Versus Application IntegrationData Distribution PatternsApplication Integration PatternsConsistency and DiscoverabilityInspiring, Motivating, and Guiding for ChangeSetting Domain BoundariesException HandlingOrganizational TransformationTeam TopologiesOrganizational PlanningWrapping Up
Data GovernanceThe Governance FrameworkProcesses: Data Governance ActivitiesMaking Governance Effective and PragmaticSupporting Services for Data GovernanceData ContractsData SecurityCurrent Siloed ApproachTrust BoundariesData Classifications and LabelsData Usage ClassificationsUnified Data SecurityIdentity ProvidersReal-World ExampleTypical Security Process FlowSecuring API-Based ArchitecturesSecuring Event-Driven ArchitecturesWrapping Up

Metadata ManagementThe Enterprise Metadata ModelPractical Example of a MetamodelData Domains and Data ProductsData ModelsData LineageOther Metadata AreasThe Metalake ArchitectureRole of the CatalogRole of the Knowledge GraphWrapping Up
Master Data Management StylesData IntegrationDesigning a Master Data Management SolutionDomain-Oriented Master Data ManagementReference DataMaster DataMDM and Data Quality as a ServiceMDM and Data CurationKnowledge ExchangeIntegrated ViewsReusable Components and Integration LogicRepublishing Data Through Integration HubsRepublishing Data Through AggregatesData Governance RecommendationsWrapping Up
The Challenges of Turning Data into ValueDomain Data StoresGranularity of Consumer-Aligned Use CasesDDSs Versus Data ProductsBest PracticesBusiness RequirementsTarget Audience and Operating ModelNonfunctional RequirementsData Pipelines and Data ModelsScoping the Role Your DDSs PlayBusiness IntelligenceSemantic LayersSelf-Service Tools and DataBest PracticesAdvanced Analytics (MLOps)Initiating a ProjectExperimentation and TrackingData EngineeringModel OperationalizationExceptionsWrapping Up
A Brief Reflection on Your Data JourneyCentralized or Decentralized?Making It RealOpportunistic Phase: Set Strategic DirectionTransformation Phase: Lay Out the FoundationOptimization Phase: Professionalize Your CapabilitiesData-Driven CultureDataOpsGovernance and LiteracyThe Role of Enterprise ArchitectsBlueprints and DiagramsModern SkillsControl and GovernanceLast Words

Content preview from Data Management at Scale, 2nd Edition

Preface

Data management is an emerging and disruptive subject. Datafication is everywhere. This transformation is happening all around us: in smartphones, TV devices, ereaders, industrial machines, self-driving cars, robots, and so on. It’s changing our lives at an accelerating speed.

As the amount of data generated skyrockets, so does its complexity. Disruptive trends like cloudification, API and ecosystem connectivity, microservices, open data, software as a service (SaaS), and new software delivery models have a tremendous effect on data management. In parallel, we see an enormous number of new applications transforming our businesses. All these trends are fragmenting the data landscape. As a result, we are seeing more point-to-point interfaces, endless discussions about data quality and ownership, and plenty of ethical and legal dilemmas regarding privacy, safety, and security. Agility, long-term stability, and clear data governance compete with the need to develop new business cases swiftly. We sorely need a clear vision for the future of data management.

This book’s perspective on data management is informed by my personal experience driving the data architecture agenda for a large enterprise as chief data architect. Executing that role showed me clearly the impact a good data strategy can have on a large organization. After leaving that company, I started working as the chief data officer for Microsoft Netherlands. In this exciting new position, I’ve worked with over 50 ...