Generative AI in the Real World: Getting Beyond the Demo with Hamel Husain

In this episode, Ben Lorica and Hamel Husain talk about how to take the next steps with artificial intelligence. Developers don’t need to build their own models—but they do need basic data skills. It’s important to look at your data, to discover your model’s weaknesses, and to use that information to develop test suites and evals that show whether your model is behaving well.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Links to Resources

1. Hamel's upcoming course on evaluating LLMs.
2. Hamel's O'Reilly publications: “AI Essentials for Tech Executives” and “What We Learned from a Year of Building with LLMs”
3. Hamel's website.

Points of Interest

• 0:00: Introduction to Hamel Husain, founder of Parlance Labs.
• 0:39: What inspired you and your coauthors to create a series on practical uses of foundation models? What gaps in existing resources did you aim to address?
• 0:56: We’re publishing “AI Essentials for Tech Executives”¹ now; last year, we published “What We Learned from a Year of Building with LLMs.”² Coming from the perspective of a machine learning engineer or data scientist—you don’t need to build or train models. You can use an API. But there are skills and practices from data science that are crucial.
• 2:16: There are core skills around data analysis and error analysis and basic data literacy that you need to get beyond a demo.
• 2:43: What are some crucial shifts in mindset that you’ve written about on your blog?
• 3:24: The phrase we keep repeating is “look at your data." What does “look at your data" mean?
• 3:51: There’s a process that you should use. Machine learning systems have a lot in common with modern AI. How do you test those? Debug them? Improve them? Look at your data; people fail on this. They do vibe checks, but they don’t really know what to do next.
• 4:56: Looking at your data helps ground everything. Look at actual logs of user interactions. If you don’t have users, generate interactions synthetically. See how your AI is behaving and write detailed notes about failure modes. Do some analysis on those notes: Categorize them. You’ll start to see patterns and your biggest failure modes. This will give you a sense of what to prioritize.
• 6:08: A lot of people are missing that. People aren’t familiar with the rich ecosystem of data tools, so they get stuck. We know that it’s crucial to sample some data and look at it.
• 7:08: It’s also important that you have the domain expert do it with the engineers. On a lot of teams, the domain expert isn’t an engineer.
• 7:44: Another thing is focusing on processes, not tools. Tools aren’t the problem—the problem is that your AI isn’t working. The tools won’t take care of it for you. There’s a process: how to debug, look at, and measure AI. Those are the main mind shifts.
• 9:32: Most people aren’t building models (pretraining); they might be doing posttraining on a base model. But there are a lot of experiments that you still have to run. There’[re] knobs you have to turn, and without the ability to do it systematically and measure, you’re just mindless[ly] turning knobs without learning much.
• 10:29: I’ve held open office hours for people to ask questions about evals. What people ask most is what to eval. There are many components. You can’t and shouldn’t test everything. You should be grounded in your actual failure modes. Prioritize your tests on that.
• 11:30: Another topic is what I call the prototype purgatory. A lot of people have great demos. The demos work, and might even be deployable. But people struggle with pulling the trigger.
• 12:15: A lot of people don’t know how to evaluate their AI systems if they don’t have any users. One way to help yourself is to generate synthetic data. Have an LLM generate realistic user inputs and brainstorm different personas and scenarios. That bootstraps you significantly towards production.
• 13:57: There’s a new open source tool that does something like this for agents. It’s called IntelAgent. It generates synthetic data that you might not come up with yourself.
• 14:36: Executives are chomping at the bit to deploy. Now you have technical, product, legal, and compliance teams in the mix. What are some of the things you’re hearing about the need to deploy and the need to be careful?
• 15:10: It’s really “Do you have a way to measure your system that you trust?" One mistake is that people use off-the-shelf eval with nice dashboards and feel like [they’re] measuring a whole bunch of stuff. But most of that won’t correlate with your most critical problems. And people don’t trust that stuff because you haven’t gone through the process.
• 16:22: These metrics are sold to executives, and in the first go-round they often get fooled. Then they learn really fast that they need metrics that they understand. That’s an alignment that happens, but people take a long time to get there.
• 17:05: I was working with a product for recruiters that helped draft letters to candidates. One of the metrics was edit distance—the less edits the better. It seemed reasonable, but when looking at the data, we found that the recruiters were making edits that made no sense or made things dramatically incorrect. That’s the kind of thing that happens when you don’t look at data.
• 18:34: Now that other teams [are] involved, the legal and technical teams don’t have a way to work together well. They communicate in writing, and each review cycle takes a few weeks.
• 20:02: We’ve been dealing with this problem in machine learning for a long time. Everyone is facing it. A lot of problems are organizational, not technical or AI: bureaucracy, not being able to communicate across teams. We talk about using plain English language wherever possible. If you say “I’m building an agent,” people don’t know what you’re doing—but the job may just be writing a prompt. Writing a prompt is within the domain of a nontechnical person.
• 22:22: It happens everywhere and alienates nontechnical people. LLMs make things accessible through English language—using jargon destroys that. Instead of saying “RAG," say “I’m going to make sure the model has the right context so it can answer questions correctly."
• 23:27: This frustrates people even within the technical world. People use different tools for different aspects of the agent stack. And everyone is trying to promote a new term.
• 24:12: A decade ago, people would say, “I’m going to use big data." You immediately knew that you didn’t know what they were talking about. It was so nebulous—almost no information in that term. “Agents" is an example of that, at least right now.
• 25:12: If someone is in school, what they need to know is different from what we had to know. Do you have any specific advice to listeners who are starting their career?
• 25:39: Learn how to use AI, integrate it into your life, use it to learn, learn how to learn effectively, learn new skills with AI as a coach. It’s hard to predict what jobs will be secure. All I know is that you have to learn new things. Practice using it every day.
• 26:57: Where do you stand on going to college, let alone grad school?
• 27:38: The cost/benefit ratio of higher education has been shrinking primarily because of the cost. To prepare yourself for college, you have to be so good that you don’t need to go to the university. You’re already a self-motivated self-learner. That’s why a lot of people drop out of college. It’s really the learning and the grit that get you a long ways.
• 29:50: But it’s hard for me to say don’t go to a four-year university.
• 30:10: Pragmatically, the value comes down to what you learn, but honestly, signaling, and the network and friendships you build in college. That’s why a lot of young people decide to quit—they’ve met some people and drop out.
• 30:47: I would like to see education change in light of AI. The way we test people for knowledge and skills doesn’t make sense, and hasn’t changed yet.
• 31:17: How else can people follow you?
• 31:30: I’m teaching a course on domain-specific evals with Shreya Shankar; she’s an amazing engineer.

Footnotes

1. With Greg Ceccarelli
2. With Eugene Yan, Bryan Bischof, Charles Frye, Jason Liu, and Shreya Shankar