The AI Pocket Book

3 Selecting and evaluating AI tools

This chapter covers

• Distinctions among different types of AI, or ways of using AI, and how to select the most appropriate one
• How to assess AI’s performance and select models
• Common ways to measure AI’s performance at a task

This chapter provides guidance on selecting an AI model or tool and assessing its performance at a given task. We kick off by discussing three common distinctions between different types of AI: proprietary versus open source AI, off-the-shelf versus fine-tuned AI, and AI apps versus foundation models. We explain what these mean and how to pick the most suitable type. Afterward, we discuss a common process to assess AI’s performance, which uses different datasets for validation and ...