Distributed Data Systems with Azure Databricks

Chapter 10: Model Tracking and Tuning in Azure Databricks

In the previous chapter, we learned how to create machine learning and deep learning models, as well as how to load datasets during distributed training in Azure Databricks. Finding the right machine learning algorithm to solve a problem using machine learning is one thing, but finding the best hyperparameters is another equally or more complex task. In this chapter, we will focus on model tuning, deployment, and control by using MLflow as a Model Repository. We will also use Hyperopt to search for the best set of hyperparameters for our models. We will implement the use of these libraries using deep learning models that have been made using the scikit-learn Python library.

More concretely, ...