Building Machine Learning Powered Applications

Chapter 2. Create a Plan

In the previous chapter, we covered how to estimate if ML is necessary, find where it could be most appropriately used, and convert a product goal to the most appropriate ML framing. In this chapter, we will cover the use of metrics to track ML and product progress and compare different ML implementations. Then, we will identify methods to build a baseline and plan modeling iterations.

I have had the unfortunate opportunity to see many ML projects be doomed from the start due to a misalignment between product metrics and model metrics. More projects fail by producing good models that aren’t helpful for a product rather than due to modeling difficulties. This is why I wanted to dedicate a chapter to metrics and planning.

We will cover tips to leverage existing resources and the constraints of your problem to build an actionable plan, which will dramatically simplify any ML project.

Let’s start with defining performance metrics in more detail.

Measuring Success

When it comes to ML, the first model we build should be the simplest model that could address a product’s needs, because generating and analyzing results is the fastest way to make progress in ML. In the previous chapter, we covered three potential approaches of increasing complexity for the ML Editor. Here they are as a reminder:

Baseline; designing heuristics based on domain knowledge

We could start with simply defining rules ourselves, based on prior knowledge of what makes for well-written ...