Redefining Data Engineering with AI

Chapter 5. Building the Data Foundation

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book. There is an active GitHub repository for this book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

So far, you have created many artifacts and spec files using various plugins discussed in previous chapters. The artifacts created were all part of the planning and decisions taken for building Patient 360. In this chapter, you will use all the artifacts to build the bronze layer of medallion architecture for loading raw data, which serves as the foundation. But if you feed all the artifacts to Claude at once, there is often a chance that the AI agent will go off track, hallucinate, suffer from context bloat, and be unable to complete the task. This chapter first addresses the gap by introducing the Scrum Master role and its plugin, which will break down work into small, atomic tasks, making the AI agent accountable. The chapter then explores the Developer role and the plugin that will implement the tasks created by the Scrum Master. By the end of this chapter, you will be able to build the Bronze layer using AI, which is the data foundation, along with the necessary ...