The Self-Service Data Roadmap

Chapter 4. Feature Store Service

So far, we have discovered the available datasets and artifacts that can be used to generate the required insight. In the case of ML models, there is an additional step of discovering features. For instance, a revenue forecasting model that needs to be trained would require the previous revenue numbers by market, product line, and so on as input. A feature is a data attribute that can be either extracted directly or derived by computing from one or more data sources—e.g., the age of a person, a coordinate emitted from a sensor, a word from a piece of text, or an aggregate value like the average number of purchases within the last hour. Historic values of the data attribute are required for using a feature in ML models.

Data scientists spend a significant amount of their time creating training datasets for ML models. Building data pipelines to generate the features for training, as well as for inference, is a significant pain point. First, data scientists have to write low-level code for accessing datastores, which requires data engineering skills. Second, the pipelines for generating these features have multiple implementations that are not always consistent—i.e., there are separate pipelines for training and inference. Third, the pipeline code is duplicated across ML projects and not reusable since it is embedded as part of the model implementation. Finally, there is no change management or governance of features. These aspects impact the overall ...