Generative AI in the Real World: Agents—The Next Step in AI with Shelby Heinecke

Join Shelby Heinecke, senior research manager at Salesforce, and Ben Lorica as they talk about agents, AI models that can take action on behalf of their users. Are they the future—or at least the hot topic for the coming year? Where are we with smaller models? And what do we need to improve the agent stack? How do you evaluate the performance of models and agents?

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2025, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Points of Interest

• 0:29: Introduction—Our guest is Shelby Heinecke, senior research manager at Salesforce.
• 0:43: The hot topic of the year is agents. Agents are increasingly capable of GUI-based interactions. Is this my imagination?
• 1:20: The research community has made tremendous progress to make this happen. We’ve made progress on function calling. We’ve trained LLMs to call the correct functions to perform tasks like sending emails. My team has built large action models that, given a task, write a plan and the API calls to execute that. This is one piece. A second piece is when you don’t know the functions a priori, giving the agent the ability to reason about images and video.
• 3:07: We released multimodal action models. They take an image and text and produce API calls. That makes navigating GUIs a reality.
• 3:34: A lot of knowledge work relies on GUI interactions. Is this just robotic process automation rebranded?
• 4:05: We’ve been automating forever. What’s special is that automation is driven by LLMs, and that combination is particularly powerful.
• 4:32: The earlier generation of RPA was very tightly scripted. With multimodal models that can see the screen, they can really understand what’s happening. Now we’re beginning to see reasoning enhanced models. Inference scaling will be important.
• 5:52: Multimodality and reasoning-enhanced models will make agents even more powerful.
• 6:00: I’m very interested in how much reasoning we can pack into a smaller model. Just this week DeepSeek also released smaller distilled versions.
• 7:08: Every month the capability of smaller models has been pushed. Smaller models right now may not compare to large models. But this year, we can push the boundaries.
• 7:38: What’s missing from the agent stack? You have the model—some notion of memory. You have tools that the agent can call. There are agent frameworks. You need monitoring, observability. Everything depends on the model’s capabilities: There’s a lot of fragmentation, and the vocabulary is still unclear. Where do agents usually fall short?
• 9:00: There’s a lot of room for improvement with function calling and multistep function calling. Earlier in the year, it was just single step. Now there’s multistep. That expands our horizons.
• 9:59: We need to think about deploying agents that solve complex tasks that take multiple steps. We will need to think more about efficiency and latency. With increased reasoning abilities, latency increases.
• 10:45: This year, we’ll see small language models and agents come together.
• 10:58: At the end of the day, this is an empirical discipline and you need to come up with your own benchmarks and eval tools. What are you doing in terms of benchmarks and eval?
• 11:36: This is the most critical piece of applied research. You’re deploying models for a purpose. You still need an evaluation set for that use case. As we work with a variety of products, we cocreate evaluation sets with our partners.
• 12:38: We’ve released the CRM benchmark. It’s open. We’ve created CRM-style datasets with CRM-type tasks. You can see the open source models and small models on these leaderboards and how they perform.
• 13:16: How big do these datasets have to be?
• 13:36: One way to approach this is to generate an evaluation set synthetically, using another LLM to generate what the data could look like.
• 14:26: There’s also the annotator route. You can pay experts, human annotators, to evaluate these sets. The key is cocreation: cocreated by model builders as well as domain experts.
• 14:55: Are transparency and explainability important for these agents?
• 15:02: We knew it was important back in the ML days. It’s especially important for agents. An agent has multiple steps, compounding. So it’s important to have a sense of what’s going on so that errors can be caught. There needs to be explainability and logging along the way.
• 16:02: Let’s go back to small models. The performance ceiling is the large model. If I can’t do something with a large model, I should temper expectations for small models unless I have a great dataset for fine tuning.
• 16:48: We’ve seen that small models can beat big models. Look at Phi-3.5. We’ve even trained a 1B function-calling model that beats models 20 times its size. Small models have smaller scope, but with the right training set they have incredible potential. Data quality is important. I’m interested to see how far we can push that.
• 17:46: One of the key lessons from DeepSeek is that reinforcement learning is becoming relevant again. Hand-labeling reasoning steps is fragile. DeepSeek shows that we can get by with reinforcement learning and supervised fine tuning.
• 18:21: DeepSeek has proven that maybe we can get by with reinforcement learning, which means we need fewer labeled datasets. It makes me hopeful that reasoning capabilities will improve given that models will be less reliant on labeled samples.
• 19:18: You’re also interested in on-device AI. What’s the latest and greatest?
• 19:27: We’ve been getting models on devices for quite a while. Now what’s enabled is on-device generative AI. Where does that fit in? 1B and sub-1B models exist. These all can go on phones. 7B is the max size we can handle on today’s phones. There are techniques we can use like quantization: We take the model, which is typically 16- or 32-bit floating point, and reduce it to 4-, 6-, or 8-bit. That really reduces the model's footprint.
• 21:18: What sort of examples on the phone are available now that people might not be aware of?
• 21:32: Right now? In terms of generative AI, I want to see more. Last year, Apple announced an on-device generative model. They were even going to have LoRA adapters to handle different tasks. I’d like to see more of that.
• 22:26: On-device AI, there are a couple of interesting dimensions. There’s hardware, of course. But frameworks are increasingly targeting the phone. Apple has MLX, which is like Apple’s version of CUDA for the phone. In any kind of new area, always look at hardware and software.
• 23:21: One framework that has gotten a lot of attention is llama.cpp. You can easily build a mobile app that can be deployed on iOS. There’s also Google’s MediaPipe, which now has generative AI support.
• 24:16: Are you paying attention to models that address audio or video?
• 24:30: My team hasn’t been thinking about those, but there’s definitely been work in the research community.
• 25:05: If you look at surveys, probably half of AI teams actually have agents in production. Where will this number be at the end of 2025?
• 25:23: This number will be higher—but the challenge will be not overhyping the technology.
• 25:57: We have some agents deployed—like Salesforce’s Agentforce. We’ll see how users use these agents, and measure the impact. With that information, it will inspire other people to deploy their agents.
• 26:35: The more people talk about success stories, the more people realize that they are using agents already without knowing it.
• 27:02: I’m sure there are plenty of apps already that have automation, but customers don’t know that it's an agent behind the scenes.