Harness Engineering for Computer Use

What you’ll learn and how you can apply it

• Understand the computer-use harness end to end: Learn how perception, grounding, action, verification, and recovery fit together as stages of one loop, and how working from pixels shapes every stage
• Build and operate a multisurface agent: Go from a provided agent harness to a working computer tool set, extend it with the shell and filesystem surfaces, route work across all three, and run a production use case on an isolated cloud sandbox
• Reason about safety, reliability, and cost: Apply sandbox isolation, confirmation gates, and prompt-injection defenses, catch silent failures with cross-surface verification, and manage the screenshot economics that govern a computer-use agent’s bill

Course description

Some of the most valuable work you can hand an agent has no API: legacy desktop applications, GUI-only vendor tools, and cross-app workflows that span a spreadsheet, a file manager, and a terminal. A computer-use agent does that work the way a person does, by seeing the screen and driving the mouse and keyboard. A capable model can already read a screenshot and choose an action, but making that reliable, safe, and affordable is the job of the harness around it, the code that captures what the model sees, executes what it picks, checks that the action landed, and blocks the actions you cannot afford.

This two-hour course opens up the computer-use agent. Sajal Sharma shows you how that harness works as you build its computer half yourself. You’ll learn what a screenshot costs in tokens, why production agents ground clicks by predicting coordinates directly, and how context management keeps a long run affordable. Then you’ll build the computer tool on a provided agent harness and drive a real cloud desktop with it, reading every step in the trace. You’ll add the desktop’s other two surfaces, the shell and the file system, and learn routing, i.e., sending each job to the cheapest reliable surface, verifying GUI work from the shell, and recovering from the silent failures that pixels alone hide. In the final segment, you’ll make it production-real with guardrails and human-in-the-loop gates for an agent that drives a whole machine, an architectural view of prompt injection, isolation, and blast radius, and a full use case run at capability, turning a folder of receipts into a report across all three surfaces. You’ll finish with how to evaluate a computer-use agent: your own task suites and the public benchmarks read critically.