Artificial Intelligence Now

Chapter 20. Building Human-Assisted AI Applications

Ben Lorica

In the August 25, 2016 episode of the O’Reilly Data Show, I spoke with Adam Marcus, cofounder and CTO of B12, a startup focused on building human-in-the-loop intelligent applications. We talked about the open source platform Orchestra for coordinating human-in-the-loop projects, the current wave of human-assisted AI applications, best practices for reviewing and scoring experts, and flash teams.

Here are some highlights from our conversation.

Orchestra: A Platform for Building Human-Assisted AI Applications

I spent a total of three years doing web-scale structured data extraction. Toward the end of that period, I started speaking with Nitesh Banta, my cofounder at B12, and we said, “Hey, it’s really awesome that you can coordinate all of these experts all over the world and give them all of these human-assisted AIs to take a first pass at work so that a lot of the labor goes away and you can use humans where they’re uniquely positioned.” But we really only managed to make a dent in data extraction and data entry. We thought that an interesting work model was emerging here, where you had human-assisted AIs and they were able to help experts do way more interesting knowledge work tasks. We’re interested, at B12, about pushing all of this work up the knowledge work stack. The first stage in this process is to build out the infrastructure to make this possible.

This is where Orchestra comes in. It’s completely open source; ...