How Agents Change an Organization: Rohit Krishnan Live with Tim O’Reilly

Engineer, economist, entrepreneur, and Wharton Generative AI Lab Senior Fellow Rohit Krishnan spends his time “building AI platforms that don’t suck,” as he puts it. Rohit’s created products and founded startups, run a hedge fund, worked as a venture capitalist, and advised companies and a government or two, so he has an extensive understanding of the ways organizations and marketplaces interact, which takes his thinking about AI capabilities down some intriguing avenues. He’s developed a pretty innovative way of testing his hypotheses: He runs experiments that put agents to work inside simulated companies, giving him the ability to control every variable to explore exactly how agents change an organization, then tracks his results on his newsletter, Strange Loop Canon. Rohit joins Tim on this episode to share some of what he’s learned from modeling human-agent coordination and where he thinks things are headed next.

Rohit will walk Tim through a few experiments that tested how well agents perform in specific real-world settings and explain what his findings suggest about how companies should organize work in the age of AI. A simulation of a service ops company showed a multi-agent workflow of five agents that by the fifth round of work produced a false company record, while an experiment on company communications (using email data from Enron of all places) demonstrated why we need to build agent institutional systems that assign agents task identities and role identities. An analysis of agent coordination illustrated that sometimes a “free for all” market where agents compete to complete tasks is more effective than having your smartest model manage other models, although another suggested that “current models can’t yet bid in a way that reflects their true capabilities.” And an investigation into using AI to audit AI reiterated the importance of implementing institutional systems around the model. Taken together, Rohit’s experiments confirm that, for now at least, agents can own “work a clear finish, limited authority and an independent check,” but humans should take the lead “when judgment determines whether the work is right.”

Tim and Rohit will also explore the challenges of “analysing pieces of software through sociological lenses” and why Rohit thinks we should start thinking of agents as “firms”; BenchBench—Rohit’s benchmark that measures how well the models can create a benchmark; what he’s learned about working with what he calls “Homo Agenticus Sapiens”; how a fondness for prehistoric animals led to some amateur paleontology “vibe science”; and other tales from the Strange Loop Canon. We hope you’ll join us for what’s sure to be a fascinating discussion. Be sure to bring your questions to help us guide it.