book

DW 2.0: The Architecture for the Next Generation of Data Warehousing

by W.H. Inmon, Derek Strauss, Genia Neushloss

July 2010

Intermediate to advanced

400 pages

9h 37m

English

Morgan Kaufmann

Read now

Unlock full access

Cover
Title Page
Copyright
Dedication
Preface
Acknowledgments
About the Authors
Table of Contents
Chapter 1: A brief history of data warehousing and first-generation data warehouses
Data Base Management SystemsOnline ApplicationsPersonal Computers and 4GL TechnologyThe Spider Web EnvironmentEvolution from the Business PerspectiveThe Data Ware House EnvironmentWhat Is a Data Warehouse?Integrating Data—a Painful ExperienceVolumes of DataA Different Development ApproachEvolution to the DW 2.0 EnvironmentThe Business Impact of the Data WarehouseVarious Components of the Data Warehouse EnvironmentThe Evolution of Data Warehousing from the Business PerspectiveOther Notions About a Data WarehouseThe Active Data Ware HouseThe Federated Data Warehouse ApproachThe Star Schema ApproachThe Data Mart Data WarehouseBuilding a “Real” Data WarehouseSummary
Chapter 2: An introduction to DW 2.0
DW 2.0—a New ParadigmDW 2.0—from the Business PerspectiveThe Life Cycle of DataReasons for the Different SectorsMetadataAccess of DataStructured Data/Unstructured DataTextual AnalyticsBlatherThe Issue of TerminologySpecific Text/General TextMetadata—a Major ComponentLocal MetadataA Foundation of TechnologyChanging Business RequirementsThe Flow of Data Within DW 2.0Volumes of DataUseful ApplicationsDW 2.0 and Referential IntegrityReporting In DW 2.0Summary

Chapter 3: DW 2.0 components—about the different sectors
The Interactive SectorThe Integrated SectorThe Near Line SectorThe Archival SectorUnstructured ProcessingFrom the Business PerspectiveSummary
Chapter 4: Metadata in DW 2.0
Reusability of Data and AnalysisMetadata In DW 2.0Active Repository/Passive RepositoryThe Active RepositoryEnterprise MetadataMetadata and the System of RecordTaxonomyInternal Taxonomies/External TaxonomiesMetadata In the Archival SectorMaintaining MetadataUsing Metadata—an ExampleFrom the End-User PerspectiveSummary
Chapter 5: Fluidity of the DW 2.0 technology infrastructure
The Technology InfrastructureRapid Business ChangesThe Treadmill of ChangeGetting Off the TreadmillReducing the Length of Time For It to RespondSemantically Temporal, Semantically Static DataSemantically Temporal DataSemantically Stable DataMixing Semantically Stable and Unstable DataSeparating Semantically Stable and Unstable DataMitigating Business ChangeCreating Snapshots of DataA Historical RecordDividing DataFrom the End-User PerspectiveSummary
Chapter 6: Methodology and approach for DW 2.0
Spiral Methodology—a Summary of Key FeaturesThe Seven Streams Approach—an OverviewEnterprise Reference Model StreamEnterprise Knowledge Coordination StreamInformation Factory Development StreamData Profiling and Mapping StreamData Correction Stream (Previously Called the Data Cleansing Stream)Infrastructure StreamTotal Information Quality Management StreamSummary
Chapter 7: Statistical processing and DW 2.0
Two Types of TransactionsUsing Statistical AnalysisThe Integrity of the ComparisonHeuristic AnalysisFreezing DataExploration ProcessingThe Frequency of AnalysisThe Exploration FacilityThe Sources for Exploration ProcessingRefreshing Exploration DataProject-Based DataData Marts and the Exploration FacilityA Backflow of DataUsing Exploration Data InternallyFrom the Perspective of the Business AnalystSummary
Chapter 8: Data models and DW 2.0
An Intellectual Road MapThe Data Model and BusinessThe Scope of IntegrationMaking the Distinction Between Granular and Summarized DataLevels of the Data ModelData Models and the Interactive SectorThe Corporate Data ModelA Transformation of ModelsData Models and Unstructured DataFrom the Perspective of the Business UserSummary
Chapter 9: Monitoring the DW 2.0 environment
Monitoring the DW 2.0 EnvironmentThe Transaction MonitorMonitoring Data QualityA Data Warehouse MonitorThe Transaction Monitor—Response TimePeak-Period ProcessingThe Etl Data Quality MonitorThe Data Warehouse MonitorDormant DataFrom the Perspective of the Business UserSummary
Chapter 10: DW 2.0 and security
Protecting Access to DataEncryptionDrawbacksThe FirewallMoving Data OfflineLimiting EncryptionA Direct DumpThe Data Warehouse MonitorSensing an AttackSecurity For Near Line DataFrom the Perspective of the Business UserSummary
Chapter 11: Time-variant data
All Data In DW 2.0—Relative To TimeTime Relativity In the Interactive SectorData Relativity Elsewhere In DW 2.0Transactions In the Integrated SectorDiscrete DataContinuous Time Span DataA Sequence of RecordsNonoverlapping RecordsBeginning and Ending a Sequence of RecordsContinuity of DataTime-Collapsed DataTime Variance In the Archival SectorFrom the Perspective of the End UserSummary
Chapter 12: The flow of data in DW 2.0
The Flow of Data Throughout the ArchitectureEntering the Interactive SectorThe Role of ETLData Flow Into the Integrated SectorData Flow Into the Near Line SectorData Flow Into the Archival SectorThe Falling Probability of Data AccessException-Based Flow of DataFrom the Perspective of the Business UserSummary
Chapter 13: ETL processing and DW 2.0
Changing States of DataWhere ETL FitsFrom Application Data to Corporate DataETL In Online ModeETL In Batch ModeSource and TargetAn ETL MappingChanging States—an ExampleMore Complex TransformationsETL and ThroughputETL and MetadataETL and An Audit TrailETL and Data QualityCreating EtlCode Creation or Parametrically Driven ETLETL and RejectsChanged Data CaptureELTFrom the Perspective of the Business UserSummary
Chapter 14: DW 2.0 and the granularity manager
The Granularity ManagerRaising the Level of GranularityFiltering DataThe Functions of the Granularity ManagerHome-Grown Versus Third-Party Granularity ManagersParallelizing the Granularity ManagerMetadata as a By-ProductFrom the Perspective of the Business UserSummary
Chapter 15: DW 2.0 and performance
Good Performance—a Cornerstone For DW 2.0Online Response TimeAnalytical Response TimeThe Flow of DataQueuesHeuristic ProcessingAnalytical Productivity and Response TimeMany Facets to PerformanceIndexingRemoving Dormant DataEnd-User EducationMonitoring the EnvironmentCapacity PlanningMetadataBatch ParallelizationParallelization for Transaction ProcessingWorkload ManagementData MartsExploration FacilitiesSeparation of Transactions Into ClassesService Level AgreementsProtecting the Interactive SectorPartitioning DataChoosing the Proper HardwareSeparating Farmers and ExplorersPhysically Group Data TogetherCheck Automatically Generated CodeFrom the Perspective of the Business UserSummary
Chapter 16: Migration
Houses and CitiesMigration In a Perfect WorldThe Perfect World Almost Never HappensAdding Components IncrementallyAdding the Archival SectorCreating Enterprise MetadataBuilding the Metadata Infrastructure“Swallowing” Source SystemsETL as a Shock AbsorberMigration to the Unstructured EnvironmentFrom the Perspective of the Business UserSummary
Chapter 17: Cost justification and DW 2.0
Is DW 2.0 Worth It?Macro-Level JustificationA Micro-Level Cost JustificationCompany B Has DW 2.0Creating New AnalysisExecuting the StepsSo How Much Does all of This Cost?Consider Company BFactoring the Cost of DW 2.0Reality of InformationThe Real Economics of DW 2.0The Time Value of InformationThe Value of IntegrationHistorical InformationFirst-Generation DW and DW 2.0—The EconomicsFrom the Perspective of the Business UserSummary
Chapter 18: Data quality in DW 2.0
The DW 2.0 Data Quality Tool SetData Profiling Tools and the Reverse-Engineered Data ModelData Model TypesData Profiling Inconsistencies Challenge Top-Down ModelingSummary
Chapter 19: DW 2.0 and unstructured data
DW 2.0 and Unstructured DataReading TextWhereto Do Textual Analytical ProcessingIntegrating TextSimple EditingStop WordsSynonym ReplacementSynonym ConcatenationHomographic ResolutionCreating ThemesExternal Glossaries/TaxonomiesStemmingAlternate SpellingsText Across LanguagesDirect SearchesIndirect SearchesTerminologySemistructured Data/Value = Name DataThe Technology Needed to Prepare the DataThe Relational Data BaseStructured/Unstructured LinkageFrom the Perspective of the Business UserSummary
Chapter 20: DW 2.0 and the system of record
Other Systems of RecordFrom the Perspective of the Business UserSummary
Chapter 21: Miscellaneous topics
Data MartsThe Convenience of a Data MartTransforming Data Mart DataMonitoring DW 2.0Moving Data from One Data Mart to AnotherBad DataA Balancing EntryResetting a ValueMaking CorrectionsThe Speed of Movement of DataData Warehouse UtilitiesSummary
Chapter 22: Processing in the DW 2.0 environment
Summary
Chapter 23: Administering the DW 2.0 environment
The Data ModelArchitectural AdministrationDefining the Moment When an Archival Sector Will Be NeededDetermining Whether the Near Line Sector Is NeededMetadata AdministrationData Base AdministrationStewardshipSystems and Technology AdministrationManagement Administration of the DW 2.0 EnvironmentPrioritization and Prioritization ConflictsBudgetScheduling and Determination of MilestonesAllocation of ResourcesManaging ConsultantsSummary
Index
Instructions for online access

Content preview from DW 2.0: The Architecture for the Next Generation of Data Warehousing

Chapter 8 Data models and DW 2.0

The world of DW 2.0 is a complex one. There are many aspects and many facets. It is very easy to become entwined in details and quickly lose your direction. It is important to keep one’s perspective when dealing with DW 2.0.

FIGURE 8.1 A data model serves as an intellectual road map.

AN INTELLECTUAL ROAD MAP

To that end, an essential part of DW 2.0 is the data model. The data model acts—in many ways—as an intellectual road map to the many parts of DW 2.0. Figure 8.1 shows the role of the data model in DW 2.0. ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Data as a Service: A Framework for Providing Reusable Enterprise Data Services

Publisher Resources

ISBN: 9780123743190

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

DW 2.0: The Architecture for the Next Generation of Data Warehousing

by W.H. Inmon, Derek Strauss, Genia Neushloss

AN INTELLECTUAL ROAD MAP

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.