Building Machine Learning Systems with a Feature Store

Chapter 4. Feature Stores

As we have seen in the first three chapters, data management is one of the most challenging aspects of building and operating AI systems. In the last chapter, we used a feature store to build our air quality forecasting system. The feature store stored the output of the feature pipelines, provided training data for the training pipeline, and provided inference data for the batch inference pipeline. The feature store is a central data platform that stores, manages, and serves features for both training and inference. It also ensures consistency between features used in training and inference, and it enables the construction of modular AI systems by providing a shared data layer and well-defined APIs to connect FTI pipelines.

In this chapter, we will dive deeper into feature stores and answer the following questions:

• What problems does the feature store solve, and when do I need one?

• What is a feature group, how does it store data, and how do I write to one?

• How do I design a data model for feature groups?

• How do I read feature data spread over many feature groups for training or inference?

We will look at how feature stores are built from a columnar store, a row-oriented store, and a vector index. We will describe how feature stores solve challenges related to feature reuse, how to manage time-series data, and how to prevent skew between FTI pipelines. And throughout the chapter, we will also weave in a motivating example of a real-time ML system that ...