Production Voice AI Agents with LiveKit

Schedule

The time frames are only estimates and may vary according to how the class is progressing.

Voice agents and the first-byte problem (15 minutes)

• Presentation: What a voice agent is and what this course covers; the STT, LLM, TTS pipeline with what plugs in at each stage (voice activity detection, semantic turn detection, function tools, retrieval, audio synthesis), where each component runs, and the course map (Hello Voice → Production UX → Grounding with Moss → Ship It)
• Demo: The finished system, a live voice call to the HeartByte orb at heartbyte.io, the same agent you’ll build, retrieving from Moss in-process

Your first voice agent (30 minutes)

• Presentation: LiveKit Agents framework fundamentals; Agent, AgentSession, JobContext, WorkerOptions, the entrypoint pattern, and how LiveKit Inference unifies STT, LLM, and TTS behind shorthand strings like “deepgram/nova-3:en”
• Hands-on exercises: Boot the dev stack and have your first voice conversation; run uv run python sections/01-hello-voice/agent.py dev, connect via the Agents Playground, speak to the agent, inspect the trace (Silero VAD → Deepgram nova-3 → GPT-4o-mini → Cartesia sonic-3); measure your first-byte latency; time the gap from your last word to first audio out, speak over the agent mid-reply, stay silent for a beat

Production UX—turn detection (15 minutes)

• Presentation: Why VAD-only silence timeouts are wrong for voice; the difference between “they stopped making sound” and “they finished their thought,” LiveKit’s semantic end-of-turn model, and why this single change is the difference between an agent that feels rude and one that feels human
• Hands-on exercise: Upgrade to semantic turn detection; swap in MultilingualModel(), test phrases with natural pauses, compare side by side to VAD-only behavior, and watch the agent wait instead of barging in
• Q&A
• Break

Grounding agents in real data with Moss (30 minutes)

• Presentation: Why voice agents need retrieval; why retrieve-as-tool patterns add an extra LLM round-trip you can’t afford in voice, how hosted vector DBs add 300 to 900 ms of dead air per turn, and the “retrieve always, let the LLM decide what to use” principle
• Hands-on exercises: Build a Moss index for the HeartByte orb; install the Moss SDK, run build_index.py against the HeartByte knowledge base (the same data file shipped at heartbyte.io), query the index directly, inspect scores and tune alpha and top_k; wire on_user_turn_completed for unconditional RAG; override the hook, query Moss on every user turn, inject results as a system message into turn_ctx, ask “What is HeartByte?” and “Tell me about Abhi’s three-phase method” and watch grounded answers in real time
• Q&A

Observability and deployment (20 minutes)

• Presentation: Observability for voice agents; per-turn latency breakdown across STT first byte, LLM time-to-first-token, TTS first audio chunk; session-level usage rollups; the principle of observability before deploy, not after
• Hands-on exercises: Wire production metrics; starting from the agent_start.py stub, subscribe to conversation_item_added and read per-turn latency from ChatMessage.metrics, subscribe to session_usage_updated for cumulative model usage, register an add_shutdown_callback for an end-of-session summary
• Demonstration: Deploy with lk agent create; one command builds, ships, and registers your agent on LiveKit Cloud’s global infrastructure; the section directory is pre-staged as a self-contained deployable project (Dockerfile on Trixie to avoid the inferedge-moss-core wheel-compat trap, pyproject.toml, uv.lock, .dockerignore, livekit.toml.example) so the build hits the LiveKit Cloud cache fast and stays inside the demo window; open the included orb starter page from orb/index.html on localhost, paste a 24-hour LiveKit token into config.js, and tap the orb to talk to your cloud-deployed agent end to end. The same three pieces students will need to ship in production: static page + LiveKit Cloud + agent worker. For subsequent agent updates, lk agent deploy reads livekit.toml and rolls a new container without touching the dashboard secrets
• Going-further callouts: Wiring the visitor-facing half on Cloudflare Pages (a static HTML page using the LiveKit JavaScript Client SDK plus a Pages Function at /api/token that mints short-lived per-visitor LiveKit tokens with the Web Crypto API, signed against your LIVEKIT_API_SECRET set as a Pages env var; the front end fetches a fresh token on every tap, never sees the secret); the three security layers a public voice page needs (an Origin / Sec-Fetch-Site check on the token endpoint, per-visitor unique room names so one bad actor’s session can’t reach another visitor’s, and Cloudflare WAF rate limiting at the edge); the audio-reactive UI state machine (idle → connecting → listening → thinking → speaking) driven from AnalyserNode taps on both the local mic and the remote agent track; multi-agent handoffs in 30 lines (TriageAgent → BillingAgent); telephony via lk sip; vision-enabled realtime models; photoreal avatars (Tavus/Anam); self-hosting LiveKit Server

Wrap-up (5 minutes)