book

Data Lakes

by Anne Laurent, Dominique Laurent, Cédrine Madera

June 2020

Beginner to intermediate

244 pages

5h 40m

English

Wiley-ISTE

Read now

Unlock full access

1.1. Introduction to data lakes1.2. Literature review and discussion1.3. The data lake challenges1.4. Data lakes versus decision-making systems1.5. Urbanization for data lakes1.6. Data lake functionalities1.7. Summary and concluding remarks
2.1. Introduction2.2. State of the art and practice2.3. System architecture2.4. Use case: the Constance system2.5. Concluding remarks
3.1. Our expectations3.2. Modeling data lake functionalities3.3. Building the knowledge base of industrial data lakes3.4. Our formalization approach3.5. Applying our approach3.6. Analysis of our first results3.7. Concluding remarks
4.1. Definitions and concepts4.2. Classification of metadata by NISO4.3. Other categories of metadata4.4. Sources of metadata4.5. Metadata classification4.6. Why metadata are needed4.7. Business value of metadata4.8. Metadata architecture4.9. Metadata management4.10. Metadata and data lakes4.11. Metadata management in data lakes4.12. Metadata and master data management4.13. Conclusion
5.1. Context5.2. Related work5.3. Metadata model5.4. Metadata implementation5.5. Concluding remarks
6.1. Introduction to master data management6.2. Deciding what to manage6.3. Why should I manage master data?6.4. What is master data management?6.5. Master data and the data lake6.6. Conclusion
7.1. Basic principles7.2. Using Linked Data in data lakes7.3. Limitations and issues7.4. The smart cities use case7.5. Take-home message
8.1. Introduction8.2. A little bit of context8.3. Every machine talks8.4. The volume paradox8.5. The fog, a shift in paradigm8.6. Constraint environment challenges8.7. Calculations and local drift8.8. Quality is everything8.9. Fog computing versus cloud computing and edge computing8.10. Concluding remarks: fog computing and data lake

9.1. Applying the notion of gravitation to information systems9.2. Impact of gravitation on the architecture of data lakes

Content preview from Data Lakes

5A Use Case of Data Lake Metadata Management

To govern a data lake with a great volume of heterogeneous types of data, metadata management is mandatory to prevent the data lake from being turned into a data swamp which is invisible, incomprehensible and inaccessible to users. In this chapter, we present a use case of data lake metadata management, applied to the health-care field, which is particularly known by its heterogeneous sources of data.

We first present a more detailed data lake definition in comparison to the chapter dedicated to the data lake definition and its underlying data lake architecture, based on which we designed the metadata model. Second, we present a metadata classification pointing to the essential attributes adapted to the use case. Third, we introduce a conceptual model of metadata which considers different types: (i) structured, (ii) semi-structured and (iii) unstructured raw or processed data. Fourth, we validate our proposition with an implementation of the conceptual model which concerns two DBMSs (one relational database and one NoSQL database).

5.1. Context

The University Hospital Center (UHC) of Toulouse is the largest hospital center in the south of France. Approximately 4,000 doctors and 12,000 hospital staff ensure more than 280,000 stays and 850,000 consultations per year. The information system of the hospital stores all the patient data including medical images, biological results, textual hospital reports, PMSI (Programme de médicalisation ...