book

Building Machine Learning Systems with a Feature Store

Name: Building Machine Learning Systems with a Feature Store
Author: Jim Dowling
ISBN: 9781098165239

by Jim Dowling

November 2025

Intermediate to advanced

508 pages

14h 13m

English

O'Reilly Media, Inc.

Read now

Unlock full access

Preface
Why Did I Write This Book?Target Readers of This BookWhat This Book Is NotOutline of the BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
I. The FTI Pipeline Architecture for Machine Learning Systems
1. Building Machine Learning Systems
The Anatomy of a Machine Learning SystemTypes of Machine LearningData SourcesMutable DataA Brief History of Machine Learning SystemsMLOps and LLMOpsA Unified Architecture for AI Systems: Feature, Training, and Inference PipelinesClasses of AI Systems with a Feature StoreML Frameworks and ML Infrastructure Used in This BookSummary
2. Machine Learning Pipelines
Building ML Systems with ML PipelinesMinimal Viable Prediction ServiceWriting Modular Code for ML PipelinesA Taxonomy for Data Transformations in ML PipelinesFeature Types and Model-Dependent TransformationsReusable Features with Model-Independent TransformationsReal-Time Features with On-Demand TransformationsThe ML Transformation Taxonomy and ML PipelinesFeature PipelinesTraining PipelinesInference PipelinesTitanic Survival as an ML System Built with ML PipelinesSummary
3. Your Friendly Neighborhood Air Quality Forecasting Service
AI System OverviewAir Quality DataExploratory Dataset AnalysisAir Quality DataWeather DataCreating and Backfilling Feature GroupsFeature PipelineTraining PipelineBatch Inference PipelineRunning the PipelinesScheduling the Pipelines as a GitHub ActionBuilding the Dashboard as a GitHub PageFunction Calling with LLMsSummary and Exercises
II. Feature Stores
4. Feature Stores
A Feature Store for Fraud PredictionBrief History of Feature StoresThe Anatomy of a Feature StoreWhen Do You Need a Feature Store?For Context and History in Real-Time ML SystemsFor Time-Series DataFor Improved Collaboration with the FTI Pipeline ArchitectureFor Governance of ML SystemsFor Discovery and Reuse of AI AssetsFor Elimination of Offline-Online Feature SkewFor Centralizing Your Data for AI in a Single PlatformFeature GroupsFeature Groups Store Untransformed Feature DataFeature Definitions and Feature GroupsWriting to Feature GroupsData Models for Feature GroupsDimension Modeling with a Credit Card Data MartReal-Time Credit Card Fraud Detection ML SystemFeature Store Data Model for InferenceOnline InferenceBatch InferenceReading Feature Data with a Feature ViewPoint-in-Time Correct Training Data with Feature ViewsOnline Inference with a Feature ViewSummary and Exercises
5. Hopsworks Feature Store
Hopsworks ProjectsStoring Files in a ProjectAccess Control Within ProjectsAccess Control at the Cluster Level Using ProjectsFeature GroupsVersioningOnline StoreOffline Store (Lakehouse Tables)Change Data Capture for Feature GroupsFeature ViewsFeature SelectionModel-Dependent TransformationsCreating Feature ViewsTraining Data as Either DataFrames or FilesBatch Inference DataOnline Inference DataFaster Queries for Feature DataSummary and Exercises
III. Data Transformations
6. Model-Independent Transformations
Source Code OrganizationFeature PipelinesData Transformations for DataFramesRow Size–Preserving TransformationsRow and Column Size–Reducing TransformationsRow/Column Size–Increasing TransformationsJoin TransformationsDAG of Feature FunctionsLazy DataFramesVectorized Compute, Multicore, and ArrowData TypesCredit Card Fraud FeaturesComposition of TransformationsSummary and Exercises

7. Model-Dependent and On-Demand Transformations
Feature TransformationsEncoding Categorical VariablesDistributions of Numerical VariablesTransforming Numerical VariablesStoring Transformed Feature Data in a Feature GroupModel-Specific TransformationsOutlier Handling MethodsImputing Missing ValuesData Cleaning as Model-Based TransformationsTarget-/Label-Dependent TransformationsExpensive Features Are Computed When NeededTokenizers and Chat Templates for LLMsTransformations in Scikit-Learn PipelinesTransformations in Feature ViewsOn-Demand TransformationsPyTorch TransformationsUsing pytestUnit TestsA Testing MethodologySummary and Exercises
8. Batch Feature Pipelines
Batch Feature PipelinesFeature Pipeline Data SourcesBatch Data SourcesStreaming Data SourcesUnstructured Data in Object Stores and FilesystemsAPI and SaaS SourcesSynthetic Credit Card Data with LLMsA Logical Model for the Data Mart and the LLMLLM Prompts to Generate the Synthetic DataBackfilling and Incremental UpdatesPolling and CDC for Incremental DataBackfill and Incremental Processing in One ProgramJob OrchestratorsModalHopsworks JobsWorkflow OrchestratorsAirflowCloud Provider Workflow OrchestratorsData ContractsData Validation with Great Expectations in HopsworksSummary and Exercises
9. Streaming and Real-Time Features
Interactive AI-Enabled Systems Need Real-Time FeaturesEvent-Streaming PlatformsShift Left or Shift Right?Shift-Right ArchitecturesShift-Left ArchitecturesWriting Streaming Feature PipelinesDataflow ProgrammingStateless and Stateful Data TransformationsApache FlinkFelderaWindowed AggregationsRolling AggregationsTime Window AggregationsChoosing the Best Window Type for AggregationsRolling Aggregations with Incremental ViewsCredit Card Fraud Streaming FeaturesASOF Joins and Composition of TransformationsLagged Features and Feature Pipelines in FelderaSummary and Exercises
IV. Training Models
10. Training Pipelines
Unstructured Data and Labels in Feature GroupsSelf-Supervised and Unsupervised LearningSupervised Learning Requires a LabelRoot and Label Feature GroupsFeature SelectionTraining DataSplitting Training DataReproducible Training DataModel TrainingModel ArchitectureCheckpoints to Recover from FailuresHyperparameter Tuning with Ray TuneDistributed Training with RayParameter-Efficient Fine-Tuning of LLMsCredit Card Fraud Model with XGBoostIdentifying Bottlenecks in Distributed TrainingModel Evaluation and Model ValidationModel Performance for Classification and RegressionModel InterpretabilityModel Bias TestsModel File Formats and the Model RegistryModel CardsSummary and Exercises
V. Inference and Agents
11. Inference Pipelines
Batch Inference PipelinesBatch Predictions for a Time RangeBatch Predictions for EntitiesScaling Batch Inference with PySparkData Modeling for Batch InferenceBatch Inference for Neural NetworksBatch Inference for LLMsOnline Inference PipelinesEnsure Offline-Online Consistency for LibrariesModel Deployments with FastAPILLM DeploymentsDeployment API for Models and Feature ViewsModel-Serving Frameworks with KServePerformance and Failure HandlingMixed-Mode UDFsNative UDFs and Log-and-WaitHandling Failures in Online Inference PipelinesModel Deployment SLOsInference with Embedded ModelsEmbedded AI-Enabled ApplicationsStream-Processing AI-Enabled ApplicationsUIs for AI-Enabled Applications in PythonSummary and Exercises
12. Agents and LLM Workflows
From LLMs to AgentsPrompt ManagementPrompt EngineeringContext WindowAgents and Workflows with LlamaIndexRetrieval-Augmented GenerationRetrieval with a Document StoreRetrieval with a Feature StoreRetrieval with a Graph DatabaseTools and Function-Calling LLMsModel Context ProtocolAgent-to-Agent (A2A) ProtocolFrom LLM Workflows to AgentsPlanningSecurity ChallengesDomain-Specific (Intermediate) RepresentationsA Development Process for AgentsAgent Deployments in HopsworksSummary and Exercises
VI. MLOps and LLMOps
13. Testing AI Systems
Offline TestingFrom Dev to ProdAutomatic Containerization and JobsEnvironments and Jobs in HopsworksModal JobsCI/CD Tests for AI SystemsFeature Pipeline TestsTraining Pipeline Tests for Model Performance and BiasTesting Model DeploymentsA/B Tests for Batch InferenceEvals for AgentsGovernanceSchematized TagsLineageVersioningAudit LogsSummary and Exercises
14. Observability and Monitoring AI Systems
Logging and Metrics for ML ModelsLogging for Batch and Online ModelsMetrics for Online ModelsMetrics for Batch ModelsMonitoring Features and ModelsData Ingestion DriftUnivariate Feature DriftMultivariate Feature DriftMonitoring Vector EmbeddingsModel Monitoring with NannyMLWhen to Retrain or Redesign a ModelLogging and Metrics for AgentsFrom Logs to Traces with AgentsError AnalysisGuardrailsOnline A/B TestingJailbreaking and Prompt InjectionLLM MetricsSummary and Exercises
15. TikTok’s Personalized Recommender: The World’s Most Valuable AI System
Introduction to RecommendersA TikTok Recommender with the Retrieval-and-Ranking ArchitectureReal-Time Personalized RecommenderFeature PipelinesTraining PipelinesOnline Inference PipelineAgentic Search for VideosThe Dirty Dozen of Fallacies of MLOpsThe Ethical Responsibilities of AI BuildersSummary
Index
About the Author

Content preview from Building Machine Learning Systems with a Feature Store

Chapter 12. Agents and LLM Workflows

LLM workflows and agents are easy to spot, as they are AI-powered services that provide a natural language API. The first LLM-powered chatbots were simple wrappers for LLMs. But they couldn’t answer questions on any event that happened after their training cutoff date. So they rapidly evolved into the complex multistep engines that can answer questions on even today’s events, using vector indexes, search engines, feature stores, and other data sources to add context information to prompts.

With the help of tools and new protocols, LLM workflows have transmogrified into agents that have a level of autonomy in how to plan and execute tasks to achieve goals. Agents are more than just LLM wrappers. They can use external tools, they have memory, and they can plan strategies to achieve goals. Agents are mostly interactive services, but there are also background agents that execute tasks autonomously, automating routine tasks such as workflow execution, process optimization, and proactive maintenance.

In this chapter, we will descend the rabbit hole of building LLM workflows and agents. We will learn the art of context engineering, providing as much context and prior knowledge as possible for every interaction with an LLM. For this, you may need to query diverse data sources (vector indexes, search engines, feature stores, etc.), call external APIs, and even use other agents. We will also introduce two protocols—Model Context Protocol (MCP) and Agent-to-Agent ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.

Read now

Unlock full access

More than 5,000 organizations count on O’Reilly

O’Reilly covers everything we've got, with content to help us build a world-class technology community, upgrade the capabilities and competencies of our teams, and improve overall team performance as well as their engagement.

Julian F.

Head of Cybersecurity

I wanted to learn C and C++, but it didn't click for me until I picked up an O'Reilly book. When I went on the O’Reilly platform, I was astonished to find all the books there, plus live events and sandboxes so you could play around with the technology.

Addison B.

Field Engineer

I’ve been on the O’Reilly platform for more than eight years. I use a couple of learning platforms, but I'm on O'Reilly more than anybody else. When you're there, you start learning. I'm never disappointed.

Amir M.

Data Platform Tech Lead

I'm always learning. So when I got on to O'Reilly, I was like a kid in a candy store. There are playlists. There are answers. There's on-demand training. It's worth its weight in gold, in terms of what it allows me to do.

Mark W.

Embedded Software Engineer

Publisher Resources

ISBN: 9781098165222Errata Page

Cloud Computing

Data Engineering

Data Science

AI & ML

Programming Languages

Software Architecture

IT/Ops

Security

Design

Business

Soft Skills

Building Machine Learning Systems with a Feature Store

by Jim Dowling

Chapter 12. Agents and LLM Workflows

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.