Data Management for Agentic Systems

Chapter 2. The Essentials of Data Management for Agentic Systems

A note for Early Release readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 2nd chapter of the final book. The GitHub repo will be made active later on.

If you’d like to be actively involved in reviewing and commenting on this draft, please reach out to the editor at @oreilly.com.

From Data Pipelines to Behavioral Substrates

Most enterprise data platforms were built around a common assumption. Data is collected, transformed, stored, and then consumed by people or systems that make decisions outside the platform. Even when machine learning entered the picture, that assumption largely held. Models were trained on historical data, deployed into bounded applications, and monitored periodically. Data management focused on feeding those models accurately and efficiently. This is the pipeline model the chapter title refers to. It has served well for decades, and certain traditional systems such as high-frequency trading, real-time fraud detection, and programmatic ad bidding have already pushed beyond parts of it. Those systems demand speed and closed-loop feedback, yet they still operate within tightly scoped, predefined logic. Agentic systems challenge the assumption more fundamentally.

An agent observes its environment, assembles ...