Machine Learning Engineering in Action

11 Model measurement and why it’s so important

This chapter covers

• Methodologies for determining the impact of a model
• A/B testing approaches for attribution data collection

In part 1, we focused on aligning ML project work to business problems. This is, after all, the most critical aspect for making the solution viable. While those earlier chapters focused on communication before, during, and immediately up to a production release, this chapter focuses on the project communication after release. We’ll cover how to present, discuss, and accurately report on the long-term health of ML projects—specifically, in language and methodologies that the business will understand.

Discussions about model performance are complex. While the business is ...