book

Data Management at Scale, 2nd Edition

by Piethein Strengholt

April 2023

Beginner to intermediate

412 pages

12h 48m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Why I Wrote This Book and Why NowWho Is This Book For?How to Read or Use This BookConventions Used in This BookO’Reilly Online LearningHow to Contact UsAcknowledgments
Recent Technology Developments and Industry TrendsData ManagementAnalytics Is Fragmenting the Data LandscapeThe Speed of Software Delivery Is ChangingThe Cloud’s Impact on Data Management Is ImmeasurablePrivacy and Security Concerns Are a Top PriorityOperational and Analytical Systems Need to Be IntegratedOrganizations Operate in Collaborative EcosystemsEnterprises Are Saddled with Outdated Data ArchitecturesThe Enterprise Data Warehouse: A Single Source of TruthThe Data Lake: A Centralized Repository for Structured and Unstructured DataThe Pain of CentralizationDefining a Data StrategyWrapping Up
Application Design Starting PointsEach Application Has a Data StoreApplications Are Always UniqueGolden SourcesThe Data Integration DilemmaApplication RolesInspirations from Software ArchitectureData DomainsDomain-Driven DesignBusiness ArchitectureDomain CharacteristicsPrinciples for Distributed and Domain-Oriented Data ManagementDesign Principles for Data DomainsBest Practices for Data ProvidersDomain Ownership ResponsibilitiesTransitioning Toward Distributed and Domain-Oriented Data ManagementWrapping Up
Domain Topologies: Managing Problem SpacesFully Federated Domain TopologyGoverned Domain TopologyPartially Federated Domain TopologyValue Chain–Aligned Domain TopologyCoarse-Grained Domain TopologyCoarse-Grained and Partially Governed Domain TopologyCentralized Domain TopologyPicking the Right TopologyLanding Zone Topologies: Managing Solution SpacesSingle Data Landing ZoneSource- and Consumer-Aligned Landing ZonesHub Data Landing ZoneMultiple Data Landing ZonesMultiple Data Management Landing ZonesPractical Landing Zones ExampleWrapping Up
What Are Data Products?Problems with Combining Code, Data, Metadata, and InfrastructureData Products as Logical EntitiesData Product Design PatternsWhat Is CQRS?Read Replicas as Data ProductsDesign Principles for Data ProductsResource-Oriented Read-Optimized DesignData Product Data Is ImmutableUsing the Ubiquitous LanguageCapture Directly from the SourceClear Interoperability StandardsNo Raw DataDon’t Conform to ConsumersMissing Values, Defaults, and Data TypesSemantic ConsistencyAtomicityCompatibilityAbstract Volatile Reference DataNew Data Means New OwnershipData Security PatternsEstablish a MetamodelAllow Self-ServiceCross-Domain RelationshipsEnterprise ConsistencyHistorization, Redeliveries, and OverwritesBusiness Capabilities with Multiple OwnersOperating ModelData Product ArchitectureHigh-Level Platform DesignCapabilities for Capturing and Onboarding DataData QualityData HistorizationSolution DesignReal-World ExampleAlignment with Storage AccountsAlignment with Data PipelinesCapabilities for Serving DataData Serving ServicesFile Manipulation ServiceDe-Identification ServiceDistributed OrchestrationIntelligent Consumption ServicesDirect Usage ConsiderationsGetting StartedWrapping Up
Introducing API ManagementWhat Is Service-Oriented Architecture?Enterprise Application IntegrationService OrchestrationService ChoreographyPublic Services and Private ServicesService Models and Canonical Data ModelsParallels with Enterprise Data Warehousing ArchitectureA Modern View of API ManagementFederated Responsibility ModelAPI GatewayAPI as a ProductComposite ServicesAPI ContractsAPI DiscoverabilityMicroservicesFunctionsService MeshMicroservice Domain BoundariesEcosystem CommunicationExperience APIsGraphQLBackend for FrontendPractical ExampleMetadata ManagementRead-Oriented APIs Serving Data ProductsWrapping Up
Introduction to EventsNotifications Versus Carried StateThe Asynchronous Communication ModelWhat Do Modern Event-Driven Architectures Look Like?Message QueuesEvent BrokersEvent Processing StylesEvent ProducersEvent ConsumersEvent Streaming PlatformsGovernance ModelEvent Stores as Data Product StoresEvent Stores as Application BackendsStreaming as the Operational BackboneGuarantees and ConsistencyConsistency LevelProcessing MethodsMessage OrderDead Letter QueueStreaming InteroperabilityGovernance and Self-ServiceWrapping Up
Cross-Domain InteroperabilityQuick RecapData Distribution Versus Application IntegrationData Distribution PatternsApplication Integration PatternsConsistency and DiscoverabilityInspiring, Motivating, and Guiding for ChangeSetting Domain BoundariesException HandlingOrganizational TransformationTeam TopologiesOrganizational PlanningWrapping Up
Data GovernanceThe Governance FrameworkProcesses: Data Governance ActivitiesMaking Governance Effective and PragmaticSupporting Services for Data GovernanceData ContractsData SecurityCurrent Siloed ApproachTrust BoundariesData Classifications and LabelsData Usage ClassificationsUnified Data SecurityIdentity ProvidersReal-World ExampleTypical Security Process FlowSecuring API-Based ArchitecturesSecuring Event-Driven ArchitecturesWrapping Up

Metadata ManagementThe Enterprise Metadata ModelPractical Example of a MetamodelData Domains and Data ProductsData ModelsData LineageOther Metadata AreasThe Metalake ArchitectureRole of the CatalogRole of the Knowledge GraphWrapping Up
Master Data Management StylesData IntegrationDesigning a Master Data Management SolutionDomain-Oriented Master Data ManagementReference DataMaster DataMDM and Data Quality as a ServiceMDM and Data CurationKnowledge ExchangeIntegrated ViewsReusable Components and Integration LogicRepublishing Data Through Integration HubsRepublishing Data Through AggregatesData Governance RecommendationsWrapping Up
The Challenges of Turning Data into ValueDomain Data StoresGranularity of Consumer-Aligned Use CasesDDSs Versus Data ProductsBest PracticesBusiness RequirementsTarget Audience and Operating ModelNonfunctional RequirementsData Pipelines and Data ModelsScoping the Role Your DDSs PlayBusiness IntelligenceSemantic LayersSelf-Service Tools and DataBest PracticesAdvanced Analytics (MLOps)Initiating a ProjectExperimentation and TrackingData EngineeringModel OperationalizationExceptionsWrapping Up
A Brief Reflection on Your Data JourneyCentralized or Decentralized?Making It RealOpportunistic Phase: Set Strategic DirectionTransformation Phase: Lay Out the FoundationOptimization Phase: Professionalize Your CapabilitiesData-Driven CultureDataOpsGovernance and LiteracyThe Role of Enterprise ArchitectsBlueprints and DiagramsModern SkillsControl and GovernanceLast Words

Content preview from Data Management at Scale, 2nd Edition

About the Author

Piethein Strengholt works as the chief data officer for Microsoft Netherlands. In this exciting role, he acts as a counterpart to CDO executives at large enterprises and is a driving force in growing the community and creating alignment with the product group. Piethein is also a prolific blogger and regularly speaks about the latest trends in data management, including the data mesh concept, data governance, and strategy at scale. He lives in the Netherlands with his family.

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Start your free trial

Data Governance: The Definitive Guide

Evren Eryurek, Uri Gilad, Valliappa Lakshmanan, Anita Kibunguchy-Grant, Jessi Ashdown

Financial Data Engineering

Tamer Khraisha

Practical Statistics for Data Scientists, 2nd Edition

Peter Bruce, Andrew Bruce, Peter Gedeck

Fundamentals of Data Engineering

Joe Reis, Matt Housley

Publisher Resources

ISBN: 9781098138851Errata Page