Voice agent quality evaluation.

The four things to measure

A voice agent has to hear, respond, act and know when to stop. Each fails independently, so each needs its own metric and its own test cases.

Choosing between voice and text channels is a separate decision; see AI chatbot versus voice agent.

Measuring word error rate properly

Word error rate is WER = (S + D + I) / N: the substitutions, deletions and insertions needed to turn the system’s transcript into the reference, divided by the reference word count. The open-source jiwer library computes WER and character error rate in Python, and NIST’s SCTK includes sclite, the long-standing tool for scoring speech recognition output against reference transcripts.

Normalisation decides the result, so fix it before you compare anything:

import re
import jiwer

def normalise(text: str) -> str:
    text = text.lower()
    text = re.sub(r"[^\w\s']", " ", text)   # drop punctuation; \w keeps macrons
    return re.sub(r"\s+", " ", text).strip()

refs = [normalise(t) for t in human_reference_transcripts]
hyps = [normalise(t) for t in agent_transcripts]
print(f"WER {jiwer.wer(refs, hyps):.3f}  CER {jiwer.cer(refs, hyps):.3f}")

Also decide how numbers are written (“twenty one” against “21”) and apply the same rule to both sides. Then measure what callers care about: entity accuracy by type. A transcript with low overall WER that gets one digit of a mobile number wrong has failed the call.

New Zealand calls need their own entity list: place names such as Whangārei, Paraparaumu and Ōtautahi, Māori surnames, street names, and phone numbers in local formats (021, 022 and 027 mobiles, 09 and 04 landlines). Build the list from your own call logs.

Turn latency and its budget

Measure turn latency as the gap between the end of the caller’s speech and the first audio the agent plays, and break it into components so you know which one to fix.

from statistics import quantiles

def turn_latencies_ms(turns: list[dict]) -> list[int]:
    return [t["agent_audio_start_ms"] - t["caller_speech_end_ms"]
            for t in turns if t.get("agent_audio_start_ms") and t.get("caller_speech_end_ms")]

def p50_p95(values: list[int]) -> tuple[float, float]:
    cuts = quantiles(values, n=100)
    return cuts[49], cuts[94]

Report p95, not just the average: the slow turns are the ones where callers say “hello?” and the conversation derails. Track talk-over rate (the caller starts speaking before the agent answers) as the behavioural signal of a latency problem. Our guide to reducing AI latency covers the model side.

Build a scenario suite

Test with three sources, each with a different job. Recorded real calls, transcribed by people, are the ground truth for hearing and task accuracy. Scripted calls made by staff with varied voices cover scenarios you have not recorded yet. Simulated callers, using generated speech and added noise, give cheap regression and load tests, but they understate real accents, hesitations and background noise, as covered in testing AI with synthetic data.

Score each scenario with explicit checks on the system of record, not just on the transcript: for reservation enquiry handling, that means the booking exists with the right party size, date and phone number. Task suites for agents are covered in AI agent evaluations.

Escalation is the safety metric

Escalation recall, the share of calls that needed a person and got one, is the number that protects callers. Label a sample of calls as “should escalate” or not, then measure:

• Recall: escalated and should have, divided by all that should have. Missed escalations are the worst failure.
• Precision: escalated and should have, divided by all escalations. Low precision wastes staff time but harms no one.
• Time to escalation: turns and seconds before transfer was offered on calls that needed it.

Do not use containment as a quality score. Pair it with repeat calls from the same number within 24 hours and the share of callers who asked for a person more than once. Workflows such as appointment enquiry routing depend on getting this right.

Speech output quality

Callers judge the agent’s voice as well as its words. For naturalness, a small listener panel rating samples on the five-point absolute category rating scale from ITU-T P.800 (excellent 5 to bad 1, averaged into a mean opinion score) is a well-established method. For pronunciation, keep a list of names the agent must say, check each one, and fix errors with a pronunciation lexicon or the substitution and phoneme elements of SSML where your speech provider supports them.

Where voice agents go wrong

Recording calls collects personal information, so tell callers that the call is recorded and handled by an AI assistant, and set retention limits. If you use callers’ voices to identify or verify them, or to infer things about them, the Biometric Processing Privacy Code 2025 applies; it lists voice as biometric information and its transition period ended on 3 August 2026. This evaluation is part of how we build and run AI voice agents.

• Hearing, latency, task completion and escalation each have a metric and a target
• WER normalisation rules (case, punctuation, numbers) are written down and applied to both sides
• Entity accuracy is measured by type, with a New Zealand names and places list
• Turn latency is logged per component and reported at p50 and p95, with talk-over rate
• The scenario suite covers corrections, noise, numbers, distress, silence and after-hours transfers
• Task success is checked in the system of record, not only in the transcript
• Escalation recall is measured on labelled calls; containment is never reported alone
• Agent pronunciation of key names is checked, and fixed with a lexicon or SSML
• Callers are told about recording and AI handling; biometric uses are assessed against the code

Written by the Levy Street team in Wellington. Last updated 19 October 2026.