Skip to Content
View all events

Harness Engineering for Browser Agents

Published by O'Reilly Media, Inc.

Intermediate content levelIntermediate

Give agents the capability to see, click, and act safely on the web

What you’ll learn and how you can apply it

  • Understand the browser harness end to end
  • Build and operate a browser harness
  • Reason about safety, reliability, and cost

Course description

Giving an agent browser access requires a harness that provides useful page observations, executes actions, and manages the state and permissions of the task. In this two-hour course, Sajal Sharma explains how these pieces fit together.

You will explore DOM, accessibility, and screenshot representations; how an agent targets page elements; and how context management affects multi-step browser tasks. An instructor-led demo shows snapshot, click, and type tools operating through Chrome DevTools Protocol. The second half covers identity and browser sessions, prompt injection and permission checks, verification, recovery, human handoff, and evaluation. A second demo shows the integration of a production browser tool. The accompanying repository provides code and instructions for further exploration.

This live event is for you because...

  • You’re an AI engineer who’s building agents, and you want to understand how the browser layer works under the hood.
  • You’re a software engineer integrating browser automation or web agents into a product and want to build, debug, and extend the harness with confidence.
  • You’re a technical builder evaluating browser-agent tools, or running a site that agents increasingly visit, and you want enough internals to choose well, operate safely, and see your pages the way an agent does.

Prerequisites

  • A Python environment—Python 3.12+ with pip/uv available to install packages
  • Node.js, to install the browser tools
  • An active Anthropic API key with access to Claude models
  • A code editor (VS Code recommended) and terminal access
  • Comfort with Python and the command line
  • A basic understanding of LLMs and AI agents (the agent loop, tools, and tool calls)
  • Familiarity with how web pages are structured (HTML and the DOM)
  • Familiarity with API keys and JSON configuration

Recommended preparation:

  • The prewired agent loop, tool stubs, and all demo code will be shared before the course

Recommended follow-up:

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Module 1: Browser Use with an Agent (60 minutes)

  • Presentation: Browser use within an agent harness; browser control; page observations and context; targeting elements and executing actions.
  • Hands-on Exercise: Building Browser Tools
  • Q&A and break

Module 2: Hardening Browser Use (60 minutes)

  • Presentation: Identity and sessions; permission checks and prompt injection; verification, recovery, and human handoff; evaluation; skills and production browser tooling.
  • Hands-on Exercise: Integrating a Production Browser Tool
  • Q&A

Your Instructor

Sajal Sharma

Sajal Sharma is an AI engineer and technology leader with over eight years of experience in AI/ML, specializing in natural language processing. He works at Liminal, a venture studio in Singapore, where he focuses on building AI-first products and shaping technology strategies. Previously, he led AI initiatives at various consulting and product companies, developing innovative AI solutions across industries. His on-demand course, Building AI Agents with LangGraph, is available on the O’Reilly learning platform. Sajal has delivered a guest lecture at Yale University and has been a mentor for Udacity and the University of Melbourne, where he guided students in machine learning and AI. He holds a master’s degree in information technology from the University of Melbourne.

Skills covered

  • AI Agents
  • AI & ML
  • Large Language Models (LLMs)