November 2025
Beginner to advanced
3h 29m
English
conference
Data Superstream: Data Engineering in the Age of AI
by Matt Housley, Lisa N. Cao, Jessica Talisman, David Talby, Scott Haines, Valliappa Lakshmanan, Mark Theunissen, Lena Hall
Closed Captioning available in German, English, Spanish, French, Italian, Japanese, Korean, Portuguese (Portugal, Brazil), Chinese (Simplified), Chinese (Traditional)
Overview
With the popularity of GenAI increasing by the day, good data engineering is more essential than ever. AI is transforming industries, but without high-quality data, even the most sophisticated models and AI agents can go awry. Data engineering is the backbone of AI workloads, as data engineers maintain and implement the infrastructure that houses data required for model training, build data pipelines that ensure accurate and accessible data, and monitor and enable model deployment.
Connect with leading-edge data engineering specialists to learn how to support AI and ML use cases, understand the emerging technologies you'll need to bridge the gap between data engineering and AI development, and pick up the latest best practices for dealing with common challenges and pitfalls.
What you’ll learn and how you can apply it
- Get an overview on how AI is bringing about a change in data engineering
- Learn how to use knowledge infrastructures and the Ontology Pipeline to build AI agents and systems
- Explore practical approaches for building streaming infrastructure and integrating AI agents using MCP
- Find out how to build scalable Gen AI pipelines with Spark NLP
- Discover the importance of open table formats, catalogs, and embedded systems as means for effective, governed Al development
- Understand the ethical considerations of working with data and AI
Recommended follow-up:
- Read Fundamentals of Data Engineering (book)
- Read AI Engineering (book)
- Read AI Agents with MCP (book)
- Take Solving Data Preparation Tasks with AI (on-demand course)
- Take Generative AI for Data Management and Remediation (on-demand course)
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Watch now
Unlock full access
More than 5,000 organizations count on O’Reilly
O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.Julian F.
I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.Addison B.
I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.Amir M.
I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.