book

Building Machine Learning Systems with a Feature Store

by Jim Dowling

November 2025

Intermediate to advanced

508 pages

14h 13m

English

O'Reilly Media, Inc.

Book available

Read now

Unlock full access

Why Did I Write This Book?Target Readers of This BookWhat This Book Is NotOutline of the BookConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
The Anatomy of a Machine Learning SystemTypes of Machine LearningData SourcesMutable DataA Brief History of Machine Learning SystemsMLOps and LLMOpsA Unified Architecture for AI Systems: Feature, Training, and Inference PipelinesClasses of AI Systems with a Feature StoreML Frameworks and ML Infrastructure Used in This BookSummary
Building ML Systems with ML PipelinesMinimal Viable Prediction ServiceWriting Modular Code for ML PipelinesA Taxonomy for Data Transformations in ML PipelinesFeature Types and Model-Dependent TransformationsReusable Features with Model-Independent TransformationsReal-Time Features with On-Demand TransformationsThe ML Transformation Taxonomy and ML PipelinesFeature PipelinesTraining PipelinesInference PipelinesTitanic Survival as an ML System Built with ML PipelinesSummary
AI System OverviewAir Quality DataExploratory Dataset AnalysisAir Quality DataWeather DataCreating and Backfilling Feature GroupsFeature PipelineTraining PipelineBatch Inference PipelineRunning the PipelinesScheduling the Pipelines as a GitHub ActionBuilding the Dashboard as a GitHub PageFunction Calling with LLMsSummary and Exercises
A Feature Store for Fraud PredictionBrief History of Feature StoresThe Anatomy of a Feature StoreWhen Do You Need a Feature Store?For Context and History in Real-Time ML SystemsFor Time-Series DataFor Improved Collaboration with the FTI Pipeline ArchitectureFor Governance of ML SystemsFor Discovery and Reuse of AI AssetsFor Elimination of Offline-Online Feature SkewFor Centralizing Your Data for AI in a Single PlatformFeature GroupsFeature Groups Store Untransformed Feature DataFeature Definitions and Feature GroupsWriting to Feature GroupsData Models for Feature GroupsDimension Modeling with a Credit Card Data MartReal-Time Credit Card Fraud Detection ML SystemFeature Store Data Model for InferenceOnline InferenceBatch InferenceReading Feature Data with a Feature ViewPoint-in-Time Correct Training Data with Feature ViewsOnline Inference with a Feature ViewSummary and Exercises
Hopsworks ProjectsStoring Files in a ProjectAccess Control Within ProjectsAccess Control at the Cluster Level Using ProjectsFeature GroupsVersioningOnline StoreOffline Store (Lakehouse Tables)Change Data Capture for Feature GroupsFeature ViewsFeature SelectionModel-Dependent TransformationsCreating Feature ViewsTraining Data as Either DataFrames or FilesBatch Inference DataOnline Inference DataFaster Queries for Feature DataSummary and Exercises
Source Code OrganizationFeature PipelinesData Transformations for DataFramesRow Size–Preserving TransformationsRow and Column Size–Reducing TransformationsRow/Column Size–Increasing TransformationsJoin TransformationsDAG of Feature FunctionsLazy DataFramesVectorized Compute, Multicore, and ArrowData TypesCredit Card Fraud FeaturesComposition of TransformationsSummary and Exercises

Feature TransformationsEncoding Categorical VariablesDistributions of Numerical VariablesTransforming Numerical VariablesStoring Transformed Feature Data in a Feature GroupModel-Specific TransformationsOutlier Handling MethodsImputing Missing ValuesData Cleaning as Model-Based TransformationsTarget-/Label-Dependent TransformationsExpensive Features Are Computed When NeededTokenizers and Chat Templates for LLMsTransformations in Scikit-Learn PipelinesTransformations in Feature ViewsOn-Demand TransformationsPyTorch TransformationsUsing pytestUnit TestsA Testing MethodologySummary and Exercises
Batch Feature PipelinesFeature Pipeline Data SourcesBatch Data SourcesStreaming Data SourcesUnstructured Data in Object Stores and FilesystemsAPI and SaaS SourcesSynthetic Credit Card Data with LLMsA Logical Model for the Data Mart and the LLMLLM Prompts to Generate the Synthetic DataBackfilling and Incremental UpdatesPolling and CDC for Incremental DataBackfill and Incremental Processing in One ProgramJob OrchestratorsModalHopsworks JobsWorkflow OrchestratorsAirflowCloud Provider Workflow OrchestratorsData ContractsData Validation with Great Expectations in HopsworksSummary and Exercises
Interactive AI-Enabled Systems Need Real-Time FeaturesEvent-Streaming PlatformsShift Left or Shift Right?Shift-Right ArchitecturesShift-Left ArchitecturesWriting Streaming Feature PipelinesDataflow ProgrammingStateless and Stateful Data TransformationsApache FlinkFelderaWindowed AggregationsRolling AggregationsTime Window AggregationsChoosing the Best Window Type for AggregationsRolling Aggregations with Incremental ViewsCredit Card Fraud Streaming FeaturesASOF Joins and Composition of TransformationsLagged Features and Feature Pipelines in FelderaSummary and Exercises
Unstructured Data and Labels in Feature GroupsSelf-Supervised and Unsupervised LearningSupervised Learning Requires a LabelRoot and Label Feature GroupsFeature SelectionTraining DataSplitting Training DataReproducible Training DataModel TrainingModel ArchitectureCheckpoints to Recover from FailuresHyperparameter Tuning with Ray TuneDistributed Training with RayParameter-Efficient Fine-Tuning of LLMsCredit Card Fraud Model with XGBoostIdentifying Bottlenecks in Distributed TrainingModel Evaluation and Model ValidationModel Performance for Classification and RegressionModel InterpretabilityModel Bias TestsModel File Formats and the Model RegistryModel CardsSummary and Exercises
Batch Inference PipelinesBatch Predictions for a Time RangeBatch Predictions for EntitiesScaling Batch Inference with PySparkData Modeling for Batch InferenceBatch Inference for Neural NetworksBatch Inference for LLMsOnline Inference PipelinesEnsure Offline-Online Consistency for LibrariesModel Deployments with FastAPILLM DeploymentsDeployment API for Models and Feature ViewsModel-Serving Frameworks with KServePerformance and Failure HandlingMixed-Mode UDFsNative UDFs and Log-and-WaitHandling Failures in Online Inference PipelinesModel Deployment SLOsInference with Embedded ModelsEmbedded AI-Enabled ApplicationsStream-Processing AI-Enabled ApplicationsUIs for AI-Enabled Applications in PythonSummary and Exercises
From LLMs to AgentsPrompt ManagementPrompt EngineeringContext WindowAgents and Workflows with LlamaIndexRetrieval-Augmented GenerationRetrieval with a Document StoreRetrieval with a Feature StoreRetrieval with a Graph DatabaseTools and Function-Calling LLMsModel Context ProtocolAgent-to-Agent (A2A) ProtocolFrom LLM Workflows to AgentsPlanningSecurity ChallengesDomain-Specific (Intermediate) RepresentationsA Development Process for AgentsAgent Deployments in HopsworksSummary and Exercises
Offline TestingFrom Dev to ProdAutomatic Containerization and JobsEnvironments and Jobs in HopsworksModal JobsCI/CD Tests for AI SystemsFeature Pipeline TestsTraining Pipeline Tests for Model Performance and BiasTesting Model DeploymentsA/B Tests for Batch InferenceEvals for AgentsGovernanceSchematized TagsLineageVersioningAudit LogsSummary and Exercises
Logging and Metrics for ML ModelsLogging for Batch and Online ModelsMetrics for Online ModelsMetrics for Batch ModelsMonitoring Features and ModelsData Ingestion DriftUnivariate Feature DriftMultivariate Feature DriftMonitoring Vector EmbeddingsModel Monitoring with NannyMLWhen to Retrain or Redesign a ModelLogging and Metrics for AgentsFrom Logs to Traces with AgentsError AnalysisGuardrailsOnline A/B TestingJailbreaking and Prompt InjectionLLM MetricsSummary and Exercises
Introduction to RecommendersA TikTok Recommender with the Retrieval-and-Ranking ArchitectureReal-Time Personalized RecommenderFeature PipelinesTraining PipelinesOnline Inference PipelineAgentic Search for VideosThe Dirty Dozen of Fallacies of MLOpsThe Ethical Responsibilities of AI BuildersSummary

Content preview from Building Machine Learning Systems with a Feature Store

Chapter 11. Inference Pipelines

Inference pipelines define the type of AI system you are building. Batch inference pipelines are batch AI systems, online inference pipelines are real-time AI systems, and agentic workflows are LLM-powered AI systems. An inference pipeline is a program that acquires inference data, applies transformations to the input data to produce one or more feature vectors, and then feeds the feature vector(s) to one or more models that output predictions. Inference pipelines can be anything from a batch/streaming/embedded program, to a network service with SLOs, to an agent that uses LLMs and tools to achieve a goal. Inference pipelines log their inputs and outputs so that you can monitor and debug their performance.

This chapter covers challenges in writing batch, online, embedded, and streaming inference programs. Agents and LLM workflows are covered in Chapter 12. You will learn how to design batch inference pipelines and scale them out with PySpark. You will learn how to write online inference pipelines that retrieve context/history from the feature store and how to deploy models in model-serving infrastructure behind a deployment API. You will learn how to embed a model in a stream-processing application and write a user interface for your AI system in Python.

Batch Inference Pipelines

Batch inference pipelines make non-time-critical predictions, run on a schedule, and output predictions to some kind of inference store, from which consumers asynchronously ...