Redefining Data Engineering with AI

While much of data engineering still means writing repetitive code, stitching together orchestration tools, and manually validating systems, the emergence of AI-assisted development is fast transforming this work. Redefining Data Engineering with AI introduces the concept of agentic engineering, a structured approach in which engineers express intent in natural language, and AI generates pipelines, documentation, test suites, and monitoring. Engineers remain in control, focusing on big-picture design, governance, and quality while delegating routine implementation to AI.

This go-to guide uses realistic case studies to show how large language models can support each phase of a data project. Through a guided build of a patient search platform, you'll learn how to design systems that integrate AI responsibly and effectively. Each chapter leads you through a practical step in the lifecycle, illustrating exactly where AI adds value and where human input is most crucial.

• Describe business goals and translate them into pipeline-ready assets
• Generate code, test cases, and documentation using structured prompts
• Integrate LLMs and governance into modern data workflows
• Automate observability and reduce operational drift with AI-driven monitoring
• Create secure data products while preserving compliance and quality
• Adopt a repeatable, intent-driven framework for scalable data practices