book

Managing Data in Motion

by April Reeve

February 2013

Beginner to intermediate

204 pages

5h 20m

English

Morgan Kaufmann

Read now

Unlock full access

What this book is about and why it’s necessaryWhat the reader will learnWho should read this bookHow this book is organizedPart 1: Introduction to data integrationPart 2: Batch data integrationPart 3: Real-time data integrationPart 4: Big data integration

The natural complexity of data interfacesThe rise of purchased vendor packagesKey enablement of big data and virtualization
Data in motionIntegrating into a common format—transforming dataMigrating data from one system to anotherMoving data around the organizationPulling information from unstructured dataMoving process to data
The differences and similarities in managing data in motion and persistent dataBatch data integrationReal-time data integrationBig data integrationData virtualization
The data integration development life cycleInclusion of business knowledge and expertise
What is batch data integration?Batch data integration life cycle
What is ETL?ProfilingExtractStagingAccess layersTransformLoad
What is data warehousing?Layers in an enterprise data warehouse architectureTypes of data to load in a data warehouse
What is data conversion?Data conversion life cycleData conversion analysisBest practice data loadingImproving source data qualityMapping to targetConfiguration dataTesting and dependenciesPrivate dataProvingEnvironments
What is data archiving?Selecting data to archiveCan the archived data be retrieved?Conforming data structures in the archiving environmentFlexible data structures
What is batch data integration architecture?Profiling toolModeling toolMetadata repositoryData movementTransformationScheduling
Why real-time data integration?Why two sets of technologies?
Interaction patternsLoose couplingHub and spokeSynchronous and asynchronous interactionRequest and replyPublish and subscribeTwo-phase commitIntegrating interaction types
Confusing terminologyEnterprise service bus (ESB)Service-oriented architecture (SOA)Extensible markup language (XML)Data replication and change data captureEnterprise application integration (EAI)Enterprise information integration (EII)
Canonical modelingMessage modeling
Introduction to master data managementReasons for a master data management solutionPurchased packages and master dataReference dataMasters and slavesExternal dataMaster data management functionalityTypes of master data management solutions—registry and data hub
Corporate information factoryOperational data storeMaster data moving to the data warehouse
What is real-time data integration metadata?ModelingProfilingMetadata repositoryEnterprise service bus—data transformation and orchestrationData movement and middlewareExternal interaction
Data integration and unstructured dataBig data, cloud data, and data virtualization
Why is data integration important in the cloud?Public cloudCloud securityCloud latencyCloud redundancy
A technology whose time has comeBusiness uses of data virtualizationData virtualization architecture
What is big data?Big data dimension—volumeBig data dimension—varietyBig data dimension—velocityTraditional big data use casesMore big data use casesLeveraging the power of big data—real-time decision supportBig data architecture
Data integration architectureData integration enginesData integration hubsMetadata managementThe end

Content preview from Managing Data in Motion

Chapter 4

The Process of Data Integration Development

Information in this Chapter

The data integration development life cycle

Inclusion of business knowledge and expertise

The data integration development life cycle

The life cycle for developing a new interface between systems is very similar to that of other data-related development disciplines. Critical to success is the analysis of the actual data on both the source and target sides of the proposed data movement. Also, although Figure 4.1 shows that the steps toward implementation and operation are sequential and distinct, they are actually more iterative and overlapping than depicted, with testing of assumptions and designs being performed as early as possible using analysis tools and prototypes. ...