# Voice agent quality evaluation.

URL: https://levystreet.com/engineering/voice-agent-quality-evaluation/
Updated: 2026-10-19

Voice AI evaluation for phone and voice agents: word error rate, entity accuracy, turn latency budgets, scenario suites and escalation precision and recall.

## Short answer

Voice AI evaluation measures whether a phone or voice agent hears callers correctly, replies fast enough to feel conversational, completes the task and hands over to a person when it should. Track word error rate and entity accuracy on your own recorded calls, turn latency from end of speech to first audio at p50 and p95, task completion, and escalation precision and recall.

- Entity accuracy on names, numbers and dates matters more than overall word error rate.
- Measure turn latency from the end of the caller's speech to the agent's first audio.
- Escalation recall is the safety metric; a high containment rate can hide trapped callers.
- Test New Zealand place names, te reo Māori and local phone formats explicitly.
- Synthetic test calls are good for regression and load but understate real accents and noise.

## The four things to measure

A voice agent has to hear, respond, act and know when to stop. Each fails independently, so each needs its own metric and its own test cases.

| Dimension | Primary metric | Definition |
|---|---|---|
| Hearing | Entity accuracy, then word error rate | Share of names, numbers, dates and references captured exactly; WER over full turns |
| Responsiveness | Turn latency, p50 and p95 | Time from the end of the caller's speech to the agent's first audio |
| Task | Task completion with correct values | Share of calls where the booking, message or answer was right in the system of record |
| Escalation | Escalation precision and recall | Of calls that should reach a person, the share that did (recall), and of transfers, the share that needed it (precision) |

Choosing between voice and text channels is a separate decision; see [AI chatbot versus voice agent](/compare/ai-chatbot-vs-voice-agent/).

## Measuring word error rate properly

Word error rate is WER = (S + D + I) / N: the substitutions, deletions and insertions needed to turn the system's transcript into the reference, divided by the reference word count. The open-source [jiwer](https://github.com/jitsi/jiwer) library computes WER and character error rate in Python, and NIST's [SCTK](https://github.com/usnistgov/SCTK) includes sclite, the long-standing tool for scoring speech recognition output against reference transcripts.

Normalisation decides the result, so fix it before you compare anything:

```python
import re
import jiwer

def normalise(text: str) -> str:
    text = text.lower()
    text = re.sub(r"[^\w\s']", " ", text)   # drop punctuation; \w keeps macrons
    return re.sub(r"\s+", " ", text).strip()

refs = [normalise(t) for t in human_reference_transcripts]
hyps = [normalise(t) for t in agent_transcripts]
print(f"WER {jiwer.wer(refs, hyps):.3f}  CER {jiwer.cer(refs, hyps):.3f}")
```

Also decide how numbers are written ("twenty one" against "21") and apply the same rule to both sides. Then measure what callers care about: entity accuracy by type. A transcript with low overall WER that gets one digit of a mobile number wrong has failed the call.

New Zealand calls need their own entity list: place names such as Whangārei, Paraparaumu and Ōtautahi, Māori surnames, street names, and phone numbers in local formats (021, 022 and 027 mobiles, 09 and 04 landlines). Build the list from your own call logs.

## Turn latency and its budget

Measure turn latency as the gap between the end of the caller's speech and the first audio the agent plays, and break it into components so you know which one to fix.

| Component | What to timestamp | Lever |
|---|---|---|
| Endpointing | Caller stops speaking, system decides the turn is over | Silence threshold: shorter cuts callers off, longer adds delay |
| Speech to text | Turn end to final transcript | Streaming recognition |
| Language model | Transcript in to first output token | Smaller model for routine turns; fewer tool round trips |
| Text to speech | First text to first audio byte | Stream speech sentence by sentence |
| Transport | Telephony and network legs | Region choice; measure from real test calls |

```python
from statistics import quantiles

def turn_latencies_ms(turns: list[dict]) -> list[int]:
    return [t["agent_audio_start_ms"] - t["caller_speech_end_ms"]
            for t in turns if t.get("agent_audio_start_ms") and t.get("caller_speech_end_ms")]

def p50_p95(values: list[int]) -> tuple[float, float]:
    cuts = quantiles(values, n=100)
    return cuts[49], cuts[94]
```

Report p95, not just the average: the slow turns are the ones where callers say "hello?" and the conversation derails. Track talk-over rate (the caller starts speaking before the agent answers) as the behavioural signal of a latency problem. Our guide to [reducing AI latency](/engineering/reducing-ai-latency/) covers the model side.

## Build a scenario suite

Test with three sources, each with a different job. Recorded real calls, transcribed by people, are the ground truth for hearing and task accuracy. Scripted calls made by staff with varied voices cover scenarios you have not recorded yet. Simulated callers, using generated speech and added noise, give cheap regression and load tests, but they understate real accents, hesitations and background noise, as covered in [testing AI with synthetic data](/engineering/testing-ai-with-synthetic-data/).

| Scenario | Example | Pass criteria |
|---|---|---|
| Happy path | "Table for four on Friday at seven" | Correct booking in the system; details read back and confirmed |
| Correction mid-turn | Caller changes the time while the agent is speaking | Agent stops talking and uses the correction |
| Noise | Call from a car, a café or with hold music | Asks the caller to repeat rather than guessing |
| Names and places | Māori surnames, Paraparaumu, Ngāruawahia | Entities captured exactly |
| Numbers | Mobile number, booking reference, date of birth | Read back and confirmed in digit groups |
| Asks for a person | "Can I talk to someone?" | Transfer offered immediately |
| Distress or complaint | Angry caller, health worry, safety issue | Escalates by policy; no improvised advice |
| Silence | Caller says nothing for ten seconds | One prompt, then a polite close, logged |
| No one available | Transfer requested after hours | Takes a message and says when someone will call back |

Score each scenario with explicit checks on the system of record, not just on the transcript: for [reservation enquiry handling](/industries/hospitality/reservation-enquiry-handling/), that means the booking exists with the right party size, date and phone number. Task suites for agents are covered in [AI agent evaluations](/engineering/ai-agent-evaluations/).

## Escalation is the safety metric

Escalation recall, the share of calls that needed a person and got one, is the number that protects callers. Label a sample of calls as "should escalate" or not, then measure:

- **Recall**: escalated and should have, divided by all that should have. Missed escalations are the worst failure.
- **Precision**: escalated and should have, divided by all escalations. Low precision wastes staff time but harms no one.
- **Time to escalation**: turns and seconds before transfer was offered on calls that needed it.

Do not use containment as a quality score. Pair it with repeat calls from the same number within 24 hours and the share of callers who asked for a person more than once. Workflows such as [appointment enquiry routing](/industries/healthcare-administration/appointment-enquiry-routing/) depend on getting this right.

## Speech output quality

Callers judge the agent's voice as well as its words. For naturalness, a small listener panel rating samples on the five-point absolute category rating scale from ITU-T [P.800](https://www.itu.int/rec/T-REC-P.800) (excellent 5 to bad 1, averaged into a mean opinion score) is a well-established method. For pronunciation, keep a list of names the agent must say, check each one, and fix errors with a pronunciation lexicon or the substitution and phoneme elements of [SSML](https://www.w3.org/TR/speech-synthesis11/) where your speech provider supports them.

## Where voice agents go wrong

| Failure | Symptom | Detection |
|---|---|---|
| Aggressive endpointing | Agent answers before the caller finishes | Interruption rate; callers repeating themselves |
| Slow turns at peak | p95 latency rises at busy times | Latency by hour; talk-over rate |
| Misheard numbers | Wrong callback numbers, failed bookings | Entity accuracy on audited calls; read-back confirmations declined |
| Trapped callers | Repeated "person please" without transfer | Escalation recall; repeat-call rate |
| Failed transfers | Transfer rings out or drops | Transfer completion rate by time of day |
| Long monologues | Agent reads lists aloud | Agent speaking time per turn |
| Unannounced recording or AI | Complaints; privacy risk | Script checks on the opening message |

Recording calls collects personal information, so tell callers that the call is recorded and handled by an AI assistant, and set retention limits. If you use callers' voices to identify or verify them, or to infer things about them, the [Biometric Processing Privacy Code 2025](https://www.privacy.org.nz/privacy-principles/codes-of-practice/biometric-processing-privacy-code/) applies; it lists voice as biometric information and its transition period ended on 3 August 2026. This evaluation is part of how we build and run [AI voice agents](/services/ai-voice-agents/).


- Hearing, latency, task completion and escalation each have a metric and a target
- WER normalisation rules (case, punctuation, numbers) are written down and applied to both sides
- Entity accuracy is measured by type, with a New Zealand names and places list
- Turn latency is logged per component and reported at p50 and p95, with talk-over rate
- The scenario suite covers corrections, noise, numbers, distress, silence and after-hours transfers
- Task success is checked in the system of record, not only in the transcript
- Escalation recall is measured on labelled calls; containment is never reported alone
- Agent pronunciation of key names is checked, and fixed with a lexicon or SSML
- Callers are told about recording and AI handling; biometric uses are assessed against the code

## Questions people ask

### How do you evaluate a voice AI agent?

Score four things separately: how accurately it hears (word error rate and accuracy on names, numbers and dates), how quickly it responds (turn latency at p50 and p95), whether it completes the caller's task correctly, and whether it escalates to a person when it should. Use recorded real calls, scripted test calls and simulated calls, and keep scoring after launch.

### What is word error rate?

Word error rate (WER) is the number of word substitutions, deletions and insertions needed to turn a transcript into the correct reference, divided by the number of words in the reference. Lower is better. It depends heavily on normalisation, such as case, punctuation and how numbers are written, so fix those rules before comparing systems.

### How fast does a voice agent need to respond?

Fast enough that callers do not talk over it or assume the line has dropped. Rather than chase a universal number, set a budget for the gap between the end of the caller's speech and the agent's first audio, measure p50 and p95 on real calls, and track how often callers start speaking again before the agent answers.

### What is a good containment rate for a voice agent?

Containment, the share of calls finished without a person, is a business outcome, not a quality score. It rises when the agent is good, and also when it traps callers who needed a human. Pair it with escalation recall on labelled calls, repeat calls from the same number within a day, and the share of callers who asked for a person more than once.

### Can voice agents handle New Zealand accents and te reo Māori names?

Modern speech recognition handles New Zealand English reasonably well, but place names, Māori surnames and local terms are where errors cluster. Build a test list of the names your callers actually use, measure entity accuracy on them, add custom vocabulary where your speech provider supports it, and check pronunciation of the agent's own speech.


## Sources

- [jiwer](https://github.com/jitsi/jiwer)
- [SCTK, the NIST Scoring Toolkit](https://github.com/usnistgov/SCTK)
- [P.800: Methods for subjective determination of transmission quality](https://www.itu.int/rec/T-REC-P.800)
- [Speech Synthesis Markup Language (SSML) Version 1.1](https://www.w3.org/TR/speech-synthesis11/)
- [Biometric Processing Privacy Code 2025](https://www.privacy.org.nz/privacy-principles/codes-of-practice/biometric-processing-privacy-code/)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.