Redefining Data Engineering with AI

Chapter 6. Building with Insight

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 6th chapter of the final book. There is an active GitHub repository for this book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Chapter 5 ended with you running counts on ingestion tables. The foundation has worked. The Bronze ingestion pipeline sourced the Synthea data from disk, a DQ suite ran against the data, and an Airflow DAG orchestrated the whole thing. We have called it the foundation. In this chapter, we will build the Silver layer on the laid foundation, with an insight into what the transforms should be to support the Patient 360 use case. The Bronze layer doesn’t answer the questions business users have; it is intended to serve only as the source of truth.

The data starts to mean something from the Silver layer. The data types are enforced; PHI may be masked per requirements; dimensions are slowly changing; and every row is expected to pass the data quality gate before it lands in the Silver layer. The data becomes more meaningful and useful from the Bronze layer. A clinician, a billing analyst, or a dashboard can query the data without knowing the underlying complexity of Medallion ...