The Self-Service Data Roadmap

Chapter 12. Model Training Service

So far, we have built the transformation pipeline for generating insights that can feed a business dashboard, or processed data for an application to share with end customers, and so on. If the insight is an ML model, model training is required; that will be covered in this chapter. A typical data scientist explores hundreds of model permutations during training to find the most accurate model. The exploration involves trying different permutations of ML algorithms, hyperparameter values, and data features. Today, the process of training ML models presents some challenges. First, with the growing dataset sizes and complicated deep learning models, training can take days and weeks. At the same time, it is nontrivial to manage training orchestration across a farm of servers consisting of a combination of CPUs and specialized hardware like GPUs. Second, iterative tuning of optimal values for model parameters and hyperparameter values relies on brute-force search. There is a need for automated model tuning, including tracking of all tuning iterations and their results. Third, for scenarios where the data is continuously changing (for instance, a product catalog, a social media feed, and so on), the model needs to be trained continuously. The ML pipelines for continuous training need to be managed in an automated fashion to continuously retrain, verify, and deploy the model without human intervention. These challenges slow down the time to train. ...