Data Superstream: Data Engineering in the Age of AI

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Introduction – Matt Housley (5 minutes)

• Matt welcomes you to the Data Superstream.

Keynote: Decentralized Data for a Centralized Brain – Valliappa Lakshmanan (30 minutes)

• Agentic AI is flipping the big data paradigm, demanding we bring the data to specialized intelligent compute, not the other way around. This shift fundamentally alters our assumptions about data modeling and storage, as LLMs perform in-context learning with vastly smaller datasets than traditional machine learning. Consequently, the growing context windows and tool calling capabilities in modern AI are quickly rendering many traditional ETL/ELT pipelines obsolete, forcing data engineers to radically rethink their entire approach. Valliappa Lakshmanan, cofounder and CTO of Obin AI, takes you through this paradigm shift, explores its future trajectory, and discusses how data engineers can adapt and thrive in this changing environment.

Designing Data Infrastructure in the Age of Generative AI – Lisa N. Cao (30 minutes)

• Developing powerful Al tooling has been the theme of the year, with agents and foundational models picking up steam across the board. But how do we serve data for agents to work effectively? What sort of interfaces and service mesh infrastructures will be required? What about at enterprise scale? What is context? Lisa N. Cao, developer relations and open source expert at Databricks, discusses the current big data landscape, the challenges to data platforming for Al, and the shifting importance of open table formats, catalogs, and embedded systems for effective, governed Al development. She’ll use open source technologies such as Apache Spark™, Unity Catalog OSS, and Apache Iceberg™ as key components of such reference architecture.

Break (5 minutes)

Knowledge Infrastructures and the Ontology Pipeline for AI Systems – Jessica Talisman (30 minutes)

• How do we build reliable, accurate AI agents and systems that scale across AI workloads? How can we engineer high-quality data pipelines delivering context-rich knowledge ecosystems for both humans and machines? Jessica Talisman, semantic architecture expert, provides some answers. Her Ontology Pipeline is a knowledge graph-based framework for designing and constructing semantic knowledge infrastructures powering AI. Its iterative logical steps—from controlled vocabularies, taxonomies, and metadata schemas to thesauri, ontologies, and knowledge graphs—harmonize data from diverse sources into structured, AI-ready assets. Discover how ontologies resolve semantic heterogeneity, boost interoperability, accelerate advanced AI analytics, and drive knowledge discovery. You’ll also explore real-world use cases and best practices for transforming unstructured and syntactic data into semantically rich, actionable assets underpinning AI workloads.

Building Scalable GenAI Inference Pipelines with Spark NLP – David Talby (30 minutes)

• David Talby, CEO of John Snow Labs and Pacific AI, looks at recent advances in the open source Spark NLP library that enable data engineers to build scalable batch GenAI inference pipelines. The emphasis is on scenarios where Spark provides a natural fit, such as processing multimodal corpora (documents, images, text) and precomputing embeddings or vision-language outputs for very large datasets. You’ll look at how Spark NLP enables efficient hardware utilization across CPUs, GPUs, and Apple Silicon, and how acceleration and quantization affect throughput in practice. You’ll also hear about integration with open-weight LLMs via Llama.cpp for private deployments, and the operational advantages of keeping ETL, enrichment, and inference within a single Spark-native workflow.

Break (5 minutes)

From Data Pipelines to AI Agents: Engineering the Future of Intelligent Systems – Scott Haines (30 minutes)

• Data engineering is evolving beyond traditional ETL pipelines. While speed and freshness remain critical, the real transformation lies in how we architect systems that don't just move data but rather enable AI agents to act on it intelligently in real-time. Software engineer Scott Haines, who specializes in distributed data systems and streaming technologies, explores practical approaches to building resilient streaming infrastructure with reduced operational overhead, and demonstrates how to integrate AI agents using Model Context Protocol (MCP) to create persistent memory streams that learn and adapt from your data flows.

The Evolving Data Engineer: Shaping AI, Shaped by AI – Lena Hall (30 minutes)

• A new era of intelligence is here. But who’ll actually build, manage, and ensure these advanced AI systems work correctly and reliably? The traditional lines of engineering are blurring, demanding skill sets that are both technically deeper and broader in scope than ever before. The engineer of tomorrow will be orchestrating complex ecosystems of specialized AI agents, rigorously validating their outputs and troubleshooting the toughest problems. Lena Hall, a senior director of developer relations at Akamai Technologies, ushers you into the future of data engineering—now inextricably entwined with AI. Join Lena to understand the evolving challenges and opportunities and how you can become a key player in building the future of intelligent systems.

Break (5 minutes)

Engineering Ethics in the Age of AI: Applying Ethical Frameworks to Data Practice – Mark Theunissen (30 minutes)

• Everyone is aware of the various moral risks that come with innovation and new technology. The emergence of big data and AI has supercharged these concerns—bringing questions of ethics, responsibility, and justice to the forefront. Educator and researcher Mark Theunissen introduces two leading ethical frameworks—responsible research and innovation (RRI) and value-sensitive design (VSD)—and considers their relevance for data engineers today. You’ll explore practical applications of these frameworks and gain insights into how engineers can navigate ethical complexity in their day-to-day work.

Closing Remarks – Matt Housley (5 minutes)

• Matt Housley closes out today’s event.