Grounded Voice Agent

An agent is only as good as what it is allowed to read. We built the ingestion side first: a pipeline that pulls transcripts from the client’s public and unlisted video catalogue and short-form social content, normalizes them into a timestamped JSONL corpus, and keeps source URLs and offsets attached to every segment.

Keeping the timestamps mattered. It means an answer can point back at the exact moment in the exact video it came from — the difference between a plausible response and a citable one.

We did not reach for a hosted vector database. The corpus is chunked into semantic units, encoded with a compact sentence-transformer model, and searched with nearest-neighbour cosine similarity. It loads in seconds, runs on the same box as the agent, costs nothing per query, and has no network hop in the latency budget.

For a corpus this size that is not a compromise — it is the correct engineering answer. Infrastructure you do not add is infrastructure you never have to pay for, secure, or explain.

The failure mode for this product is an agent that answers from the model’s general knowledge and sounds like generic advice. So retrieval is not optional. The agent is instructed to call content search on every question before it speaks, to answer from what comes back, and never to claim it has no information when the retriever returned results.

Around that core, it carries a small set of tools with clear boundaries: semantic search over the library, live web search for current facts outside the corpus, and email for follow-up. Each one exists because a real conversation needed it.

Text-in, text-out was never the product. The agent runs on a real-time agents framework with a streaming multimodal model, cloud noise cancellation, and optional video input. It can be interrupted mid-sentence, which is the single feature that separates a voice agent from a phone tree.

Responses are held short and direct by design. Spoken answers have a different length budget than written ones, and most voice assistants fail by ignoring that.