AI Essentials for Tech Executives

Chapter 8. Looking at Data: Our Your Secret Weapon

We’ve touched on the importance of having metrics that are specific to your business and a systematic process for measuring and reviewing them.

In this chapter, I want to share a recent experience that highlights a crucial lesson for AI projects:

There’s no substitute for examining your data firsthand.

I was recently working with a company that automates HR functions like recruiting and onboarding. Their engineering team had developed an evaluation suite with various metrics to measure the AI’s performance. One metric in particular caught my attention: the edit distance between the AI-generated email and the recruiter’s final version. In case you haven’t heard of it, edit distance is a measurement of how similar two texts are. This metric seemed like it would be a good one—it’s business-specific, and it can be systematically measured.

The team had found that the average edit distance between the AI-generated emails and the recruiter’s final version was 12%. This seemed like a good result, but the team was struggling with user adoption. It turned out that the metric was hiding a critical flaw.

Look at the Raw Data

In my experience, the best way to understand a metric is to look at the raw data. It might sound simple, but it’s a secret weapon that almost always uncovers something unexpected.

I asked to review some of these emails myself, and what I found was shocking: for the most part, the AI-generated emails were perfectly reasonable. ...