Redefining Data Engineering with AI

Chapter 2. Understanding AI Evolution and Tools

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. There is an active GitHub repository for this book.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at [e-mail].

Following Google’s invention of the Transformer architecture in 2017, numerous advancements have led to current AI capabilities. AI has become the most powerful tool, and it should be part of your arsenal. In data engineering, AI is helpful if you’re dealing with messy logs, slow pipelines, and exploding storage costs. AI can automate decisions, predict failures, or suggest the right schema, and much more. By the end of this chapter, you’ll know the most commonly used AI tools today, understand their scope and differences, and learn how they’re changing the game for data teams.

The term LLM didn’t gain prominence until 2018, and it refers not to a single architecture but to a class of models. The Transformer architecture was flawed with limited intelligence, and researchers began exploring how much they could scale it, a pattern often described as the “Scaling Law.” Researchers found that performance improved dramatically when neural networks (the internals ...