Building Data Integration Solutions

Chapter 2. Key Concepts in Data Integration

There are a plethora of labels and terms within the ontology of data integration. Some of the terms are synonymous and are used interchangeably. The majority, however, describe concepts that serve unique purposes within the broader data ecosystem. It is beneficial for data engineers to understand the distinct roles various concepts play and how they relate to one another in order to effectively architect and implement data integration strategies and solutions.

Note

To precisely communicate these topics necessitates a clear understanding of terms, many of which are often—for whatever reason—intentionally left vague. Therefore, in this chapter, for the more ubiquitous terms and certain terms often burdened with multiple interpretations, I attempt to clarify what I mean when I use them.

The first group of concepts involves characteristics of data and datasets, including data types, structure types, metadata, data orientations, file formats, and character encodings. The second group of concepts relates to static data and storage. The third group includes concepts related to data movement and transformations. Finally, the fourth group includes a handful of data integration management topics.

Data Properties

There are different types of data, various data file formats and format categories, multiple character encodings that computers use to interpret and handle data internally, and a slew of other characteristics about data with which you ...