Scaling Machine Learning with Spark

Chapter 3. Managing the Machine Learning Experiment Lifecycle with MLflow

Machine learning development and data science are often done in collaboration—but building models collaboratively, while experimenting with a large combination of features, standardization techniques, and hyperparameters, is a complex undertaking. Part of the reason for this is simply because it’s a complex task to track experiments, reproduce results, package models for deployment, and store and manage those models in a way that ensures they are well documented and provide the desired accuracy.

To facilitate this process, there is a need to evolve the machine learning development workflow so it is more robust, predictable, and standardized. To this end, many organizations have started to build internal machine learning platforms to manage the machine learning lifecycle. However, these platforms often support only a small set of built-in algorithms, dictated by the company’s infrastructure and the software that is available, without much openness to supporting new software due to the added complexity. Moreover, these platforms are usually not open source, and users cannot easily leverage new machine learning libraries or share their work with others in the community.

Managing the machine learning experiment lifecycle, sometimes referred to as MLOps, is a combination of machine learning, development, and operations. The machine learning part is all about the experiment itself: training, tuning, and finding ...