Enterprise AI in the Cloud

17Training, Tuning, and Evaluating Models

Success is not final; failure is not fatal: It is the courage to continue that counts.

—Winston Churchill

Traveling the AI journey is like solving a jigsaw puzzle. In the previous chapter, we got one step closer to solving this puzzle by learning how to choose the most apt AI/ML algorithm, a critical step in transitioning data into actionable intelligence. This chapter builds that up to get into the actual act of modeling, which brings these algorithms to life.

This chapter delves into the three pillars of modeling: training, tuning, and evaluation (see Figure 17.1). The focus is on ensuring these models are secure, efficient, and high performing. We begin by looking at the intricacies of model building and the likely challenges during the training process. From distributed processing to container code, we dive into the tools and techniques of model development. You learn about optimizing models for high performance using hyperparameters and model tuning. Evaluation and validation will help you align with your enterprise objectives when dealing with real-world data and ensure the models are robust and ready to achieve tangible, transformative outcomes.

MODEL BUILDING

Model building is an iterative process, and it is the first step in the process that sets up the model for training to begin. Figure 17.2 shows the model development lifecycle.

The purpose of model building is to build a working model that can be used to make predictions ...