Manthan

How to Evaluate Voice Agents

· Manthan Gupta

A voice agent can pass every eval you wrote for it and still be broken in production. The transcript reads correctly, the tool call fires with the right arguments, the backend state lands where it should, and the scorecard marks the call a success. Then you listen to the actual recording and hear the agent talking over the caller, mangling their name, and leaving three seconds of dead air before it confirms anything.

I have watched that gap open up more than once, and the cause is always the same: the agent was evaluated like a text agent with a microphone attached. A voice agent is a real-time distributed system, and most of its product quality lives in dimensions a transcript simply cannot represent, like when the agent started speaking, how long the user waited, whether it yielded when interrupted, and how it sounded while doing it. An eval that only reads transcripts after the call is grading a summary of the interaction rather than the interaction itself.

When I wrote the voice agents primer, I listed evaluation as one of the things standard LLM evals completely miss and left it for a future post. A complete evaluation has to cover the call itself: choosing the right unit of evaluation, scoring the four layers of a call, working out whether your simulator and LLM judge can be trusted, and turning the results into a ship-or-don’t-ship decision.

This is the third and final part of the voice agents series, following the voice agents primer and voice agents memory.

Let’s get into it.

The Unit of Evaluation Is the Call

The naive eval stack converts the audio to text, passes that text to the LLM, and scores the response. This measures whether the agent produced a reasonable answer after transcription, but misses whether it waited too long, interrupted the user, ignored a barge-in, misheard a critical entity, sounded bored, spoke over background noise, or completed the backend action correctly.

The modern voice eval stack evaluates the live call path rather than an isolated response. This is more representative of product quality because the goal is a voice agent that feels as natural and helpful to the user as possible. Across voice eval tools, the common pattern is:

  1. Define a scenario.
  2. Run a real or simulated call.
  3. Capture the full trace.
  4. Score behavior with multiple evaluators.
  5. Turn failures into regression tests.

A Real Eval Scenario

Consider an appointment scheduling voice agent for a clinic. A weak eval would look like this:

User: I need to move my appointment to Friday afternoon.
Expected: Agent reschedules appointment.

For a text agent, that eval may be enough to check logical correctness. For a voice agent, it omits timing, audio, turn-taking, tool execution, and business outcome, all of which contribute to product quality.

A useful eval scenario has a caller persona, audio conditions, expected tool behavior, success criteria, and failure attribution:

scenario: reschedule_existing_appointment
persona:
  name: "Impatient returning patient"
  speaking_style: "fast, interrupts once, says the date ambiguously"
  environment: "car bluetooth with mild road noise"

initial_state:
  patient_id: "p_123"
  existing_appointment: "2026-05-27 10:30"
  available_slots:
    - "2026-05-29 14:00"
    - "2026-05-29 16:30"

caller_goal:
  reschedule appointment to Friday afternoon

required_behavior:
  - verify patient identity before changing appointment
  - clarify which Friday if date is ambiguous
  - offer available afternoon slots
  - call reschedule_appointment with the selected slot
  - confirm the new appointment time out loud

failure_modes_to_track:
  - missed identity verification
  - wrong date resolution
  - wrong tool arguments
  - ignored interruption
  - response latency over 900ms p95
  - user repeats same information twice

A Passing Simulation Proves Almost Nothing

Voice agents are stochastic twice over: the model samples different responses, and borderline ASR or turn-detection decisions can change with small variations in audio. One successful call proves possibility, not reliability. Run each important scenario repeatedly and report a rate.

This is why EVA-Bench reports both pass@k and pass^k. pass@k asks whether at least one of k runs succeeds, which measures the system’s ceiling. pass^k estimates the probability that k independent runs would all succeed, which measures consistency. A system with high pass@3 and low pass^3 can produce an impressive demo, but it is not dependable enough for a workflow where every call matters.

The simulator also needs an eval. Synthetic callers are useful because they scale, but they are not automatically representative of humans. They may speak too cleanly, cooperate with clarifying questions, avoid real disfluencies, or share model family quirks with the agent under test. Validate simulator behavior against a held out set of human calls: compare task paths, interruption patterns, turn lengths, failure categories, and pass rates. If improvements on synthetic calls do not predict improvements on real calls, you are optimizing against the simulator.

Layer 1: Did It Hear the User?

ASR (automatic speech recognition) evaluation usually starts with Word Error Rate, and WER is still useful. If you swap STT (speech-to-text) providers and your domain WER jumps from 7% to 14%, you probably made the agent worse. Its business impact, however, depends on which words were transcribed incorrectly.

Substituting “a” for “the” may be harmless. Hearing “May fifteenth” as “May fifty,” dropping the word “not,” mangling a medication name, or capturing “cancel my subscription” as “cancel my description” can break the call. A single word can be the whole task.

For production voice agents, I would track three ASR metrics:

Domain WER measures transcription quality on your actual vocabulary: product names, customer names, addresses, appointment types, insurance terms, SKUs, airport codes, whatever your domain contains. Build this from real calls.

Entity error rate measures whether the fields that drive business logic were captured correctly: names, dates, amounts, confirmation numbers, addresses, and IDs. This is often more predictive than WER. A transcript can be 95% correct and still fail the call because the 5% contained the appointment time.

Streaming stability measures how much partial transcripts change before finalization. If your agent starts planning on partial ASR, final WER is too late. You need to know whether the intermediate transcript was stable enough to feed into turn detection, early intent classification, or speculative tool planning.

The subtle bug here is attribution. If ASR turns “Friday at four” into “Friday at floor,” the LLM may ask a weird clarification. A transcript-only eval misattributes this as LLM confusion; the ASR handed the model bad input.

Layer 2: Did It Manage the Conversation?

This is the part text evals completely miss.

Voice is continuous time. Users pause, breathe, backchannel, self-correct, talk over the agent, and ask a different question halfway through the answer. A good voice agent needs both answer quality and conversation control. Building the right barge-in behavior is only half the job; you also have to measure whether it works across a range of real conditions.

The research world has started catching up here. Full-Duplex-Bench evaluates pause handling, backchanneling, turn-taking, and user interruption. Its v1.5 overlap scenarios include interruption, backchannel, talking to others, and background speech, then measure whether the model stops, responds, resumes, or incorrectly holds the floor. The timing metrics are exactly the kind of thing production teams should integrate in their evals from the Full-Duplex-Bench. Stop latency when interrupted, response latency after interruption, and whether the model actually took the new user intent.

This matters because “interruption count” alone is ambiguous. A high interruption count can mean users are engaged and naturally backchanneling. It can also mean the agent is too verbose and users keep cutting it off. You need to track these categories:

End-of-speech accuracy: Did the agent detect that the user was done, or did it jump in during a pause?

Yield latency: When the user interrupted, how long did the agent keep speaking?

Backchannel selectivity: Did the agent ignore “mm-hmm” and “yeah” when they were acknowledgements, but stop for “wait, actually…”?

Agent interruption rate: How often did the agent begin speaking before the user had finished?

Silence budget: How much dead air did the user experience across the whole call, especially at p90 and p99?

This is where a lot of “the model is bad” complaints are actually VAD, endpointing, or turn detection complaints. If the endpointer fires too early, the agent responds to half a sentence. If it fires too late, the call feels sluggish. If barge-in is too sensitive, the agent stops for coughs and background TV. If it’s too conservative, the user feels trapped.

Measure the Latency Waterfall, Not Just the Total

End-to-end latency measures how long the user waited, while tracing each pipeline stage identifies the cause. Trace a single turn from the moment the user stops speaking, to the endpointer deciding the turn is actually over, to a stable ASR transcript, to the LLM’s first token (or, on a tool turn, the completed tool call, the tool’s own execution, and the model’s second pass afterward), to the first frame of synthesized audio, and finally through the network buffer and playback until the user actually hears something. In practice these stages overlap rather than run in a neat line, which is exactly why you want them captured as trace spans instead of a single number.

The one metric that represents the product is end-of-turn time to first audio (TTFA), measured from the user finishing their turn to the first audible response reaching them, and you want it at p50, p95, and p99. Everything else in the trace is a diagnostic. A slow call might come from conservative endpointing, unstable partial transcripts, the LLM’s time to first token, tool-call generation, a tool’s tail latency, the post-tool model pass, TTS startup, or the jitter buffer, and until you can see the breakdown you can’t tell which one to fix. Without it, teams tend to optimize whichever component is easiest to see on a dashboard rather than the one actually costing the user their time.

Tool-using turns need special treatment because they can contain two silences: one before an acknowledgement and another while the tool runs. Measure both. A fast first acknowledgement can make the interaction feel responsive even when the backend operation takes longer, but it must not become fake reassurance before the tool has succeeded.

Layer 3: Did It Think and Act Correctly?

Once the agent has heard the user and managed the turn, evaluate reasoning and action through four checks.

Intent and state tracking: Did the agent understand what the user wanted, and did it preserve the relevant state across turns? In voice, the same intent often arrives over multiple messy fragments: “I need to move it… no not tomorrow… the Friday one… afternoon if you have it.” The eval should verify the final state, not just each turn.

Tool-call correctness: Did the agent call the right tool at the right time with the right arguments? This should be structured, not judged from prose.

Business outcome: Did the backend state actually change correctly? Success means the appointment moved, the refund was issued, the ticket was created, or the transfer reached the right queue, not merely that the agent said “you’re all set.”

Policy and safety: Did the agent follow the rules that matter in your domain? Healthcare, finance, insurance, and recruiting agents all have steps that can’t be “mostly right.” Identity verification, consent, disclosure, escalation, and PII handling need hard pass/fail assertions.

This is where LLM-as-judge is useful, but only if the rubric is specific. “Was the agent helpful?” is weak. “Did the agent verify identity before revealing appointment details?” is strong. “Did the agent explain that cancellation fees apply before completing the cancellation?” is strong. “Did the agent avoid promising a refund before checking eligibility?” is strong.

Good judges are boring. They read like QA checklists.

Your Judge Needs an Eval Too

An LLM judge is another model in the system, not ground truth. Before using one as a release gate, build a human-labeled calibration set from real calls and measure agreement for each rubric dimension. Track false passes separately from false failures: a judge that occasionally underrates politeness is annoying but a judge that misses policy violations is dangerous.

For subjective comparisons such as naturalness or concision, pairwise judgments are often easier to calibrate than absolute scores. Run both A/B and B/A orderings because judges can systematically prefer one presentation position. Pin the judge model, prompt, rubric, and version alongside every score, then rerun calibration when any of them changes. Recent large-scale work on judge reliability found that high test-retest consistency can coexist with meaningful position bias, which is exactly why “the judge gives stable scores” is not enough validation.

Structured facts should not go through an LLM judge at all. Tool name, arguments, ordering, backend state, latency, and explicit verification or disclosure events should be checked directly when the trace represents them. If compliance exists only in the conversation’s semantics, use a calibrated judge or human review. Save model judges for dimensions that genuinely require interpretation.

Layer 4: Did It Sound Good?

TTS quality is still weirdly under evaluated in voice products. Teams will A/B test three LLM prompts for a week and then pick a voice because it sounds acceptable in the dashboard preview. That’s not enough once calls get long.

The generated audio needs its own evaluation pass. Start with the obvious checks: clipping, dropouts, volume swings, weird pauses, repeated audio loops, pronunciation failures, and latency to first audio. Once those are covered, you still need an automated score for naturalness, because nobody can listen to every call. The speech research community has a few models that try to predict how a human would rate a clip, and they are useful as filters even if they are not perfect judges.

The shared idea behind most of them is MOS, mean opinion score. In classic listening tests, people rate speech on a 1 to 5 scale from bad to excellent, and the average of those ratings becomes the MOS. Listening tests do not scale, so teams use neural models trained to approximate that human score from the audio itself.

UTMOS is one of those predictors, built specifically for synthesized speech. You give it the agent’s TTS audio and it returns a MOS-like naturalness score. Think of it as a quick answer to “does this voice sound human, or does it sound like a robot reading from a script?” It is useful for comparing voices and spotting regressions after a TTS change, but it will not tell you whether a phone number was pronounced wrong.

NISQA and its TTS variant NISQA-TTS go further by scoring speech quality without needing a clean reference recording to compare against. That matters in production because you usually only have the call that went out, not a perfect original. Beyond a single overall score, NISQA breaks quality into dimensions like noisiness, coloration, discontinuity, and loudness, which helps you separate “the voice sounds robotic” from “the call path is mangling the audio.” Use it when you want more than a naturalness number and need a hint about why the audio feels bad.

DNSMOS is aimed at a different problem: audio that has already been through noise suppression, telephony codecs, or otherwise degraded channels. If your agent runs over phone lines, Bluetooth, or a noisy room, DNSMOS is often a better fit than a model trained mostly on clean studio speech. It helps answer whether the delivery path is hurting intelligibility, not just whether the TTS voice itself is pleasant.

Reverse ASR is the blunt, practical check for the thing those naturalness models miss. Take the text the agent intended to say, synthesize it with TTS, then run a strong speech-to-text model over the resulting audio and compare the transcript back to the original text. If the agent meant to say “May fifteenth” and the round trip comes back as “May fifty,” the TTS pronounced it wrong even if the clip still scored well on naturalness. This is crude, but it catches word-level failures that MOS predictors routinely miss.

The frontier is moving toward learned speech judges. SpeechJudge builds a human preference dataset of speech pairs and trains a generative reward model for naturalness judgment. Its results show that even strong audio models still struggle to match human naturalness judgments: SpeechJudge-GRM reaches 77.2% accuracy on its benchmark, rising to 79.4% with inference-time scaling.

So use automated audio metrics as filters, not final truth. The last mile still needs humans listening to real calls.

The Harness I Would Build

If I were building voice-agent evals from scratch, I would start with five datasets.

The first is a golden scenario set: 50-100 hand-written scenarios covering your core user journeys. Each scenario has a persona, initial backend state, expected tool calls, expected final state, and policy requirements. This is your CI suite.

The second is a real-call regression set: production calls that failed, got escalated, received bad CSAT, or triggered manual review. Preserve the audio, timing, transcript, tool trace, and outcome. This is the most valuable dataset you will own.

The third is a perturbation and resilience set: the same core scenarios under accents, background noise, fast speech, long pauses, interruptions, far-field audio, telephony codecs, and Bluetooth style degradation, plus packet loss, jitter, reconnects, tool timeouts, delayed webhooks, and partial backend failures. It should verify that retries do not duplicate bookings or payments and that an interrupted call cannot leave the backend in an ambiguous state.

The fourth is a held-out release set: representative scenarios that prompt authors, simulator tuning, and judge calibration never use directly. Regression suites eventually become something teams optimize against. A private release set tells you whether the new build actually generalized or merely learned the tests. Because repeated release decisions still leak information through model selection, rotate this set periodically and keep a smaller untouched audit set for major releases.

The fifth is a human audit sample: 10-20 calls per week that someone actually listens to. Not reads. Listens. The ear catches awkwardness that transcripts hide, and those reviews provide the labels needed to keep simulators and automated judges honest.

For every run, store the same trace shape. Group it by what you will eventually need to debug:

Category What to store Why it exists
Identity call_id, scenario_id, build and provider versions so you can rerun the exact same system later
Audio inbound and outbound audio, TTS chunks with timestamps so humans and audio judges can listen to the real call
Hearing partial and final ASR transcripts, entity extractions, confidence so you can tell whether the model was confused or handed garbage
Turn-taking VAD events, endpointing decisions, interruption marks so you can measure yield latency and barge-in behavior
Reasoning LLM inputs and outputs, tool calls and results so you can assert intent, arguments, and policy steps
Timing latency timeline, transport events, retries so a slow call decomposes into a waterfall instead of a single number
Outcome final backend state, judge scores, human review notes so “you’re all set” is checked against what actually changed

That trace is also a privacy liability. Raw audio may contain biometrics, payment details, health information, addresses, and anything else a caller says aloud. Treat audio, unredacted transcripts, redacted transcripts, metadata, and aggregate metrics as separate retention classes. Redact before broad search or analytics, restrict access to raw traces, and record which third-party STT, LLM, TTS, and judge services received the data. The eval system should not become the least governed copy of your production calls.

Layer What to Score Example Gate
ASR entity error rate, domain WER, partial stability critical entities correct
Turn-taking endpointing, yield latency, backchannel selectivity yield within 500ms on interruption
Agent reasoning intent, state tracking, policy compliance required verification completed
Tools tool name, arguments, ordering, final state backend state matches expected
Audio output TTFB, naturalness, artifacts, pronunciation no clipping/dropouts, terms pronounced correctly
Outcome task success, escalation correctness, user sentiment scenario passes business goal

Turn Scores Into a Release Decision

Don’t average every metric into one reassuring number. Policy, safety, identity verification, required disclosures, tool arguments, and final backend state all need hard gates: one severe violation fails the scenario regardless of how good the latency and naturalness look. Use thresholds for latency, ASR, and audio artifacts, then reserve weighted or preference scores for softer qualities such as tone and concision.

Run critical scenarios multiple times and block on reliability, not the best attempt. Report the pass rate with its sample count and confidence interval; a 90% result over ten calls is not the same evidence as 90% over a thousand. Track flake rate explicitly, because a build that becomes demonstrably more reliable may be more valuable than one that slightly improves the average judge score.

Slice every important metric by language, accent, device, codec, region, call length, intent, and customer cohort. Aggregate WER, latency, or task success can hide failures concentrated in one of these groups. Release reports should show both the overall number and sufficiently sampled worst-performing slices, with sample counts and uncertainty. Users experience their slice, not your average. The useful metrics are the ones that point to action.

If entity error rate fails, inspect ASR and domain vocabulary. If yield latency fails, inspect VAD, AEC, barge-in, and audio playback cancellation. If tool arguments fail, inspect state tracking and schema design. If naturalness fails, inspect TTS voice, text chunking, pronunciation dictionaries, and response length. If task success fails but all component metrics pass, your scenario design or business workflow is probably wrong.

The Production Loop

When a real call fails, do not merely tag it in a dashboard. Replay fixed audio where the conversational path can remain fixed, such as ASR, audio, and component tests. For end-to-end regression, preserve the failure’s conditions and intent in an adaptive scenario so the synthetic caller can respond when a changed agent takes a different path.

In practice the loop is continuous. You ship a build, monitor real calls, find a failure, label the root cause, turn that failure into a regression test, fix the underlying issue, rerun the suite, and only then ship again. The regression is the part most teams skip. Without it, the same failure can reappear three releases later and look like a new bug.

Over time, your eval set becomes a map of every strange thing users actually do: the person calling from a car, the user who interrupts every sentence, the caller who says “yeah” as a backchannel, the customer with a noisy TV in the background, the patient who gives a date without a year, the angry user who refuses to follow the happy path.

Regression sets need maintenance. Deduplicate near-identical failures, preserve the original raw case and its normalized or adaptive test, version scenarios and rubrics, and keep the held-out release set separate from day-to-day prompt tuning. Otherwise the suite grows slower while the team gradually overfits to a museum of old incidents.

That maintained dataset is more valuable than any generic benchmark.

Parting Words

Voice eval is hard because the product is the whole live interaction: what the user said, what the system heard, when the agent responded, how it sounded, whether it yielded, which tools it called, and what changed in the real world.

So evaluate the call, not the text.

If you found this interesting, I’d love to hear your thoughts. Share it on Twitter, LinkedIn, or reach out at guptaamanthan01[at]gmail[dot]com.

References