Short answer
LLM evaluation is measuring how well a language-model system does its specific job, using a fixed set of real test cases, clear scoring rules and repeatable runs. Levy Street builds evaluation suites from your own documents and tickets, combining exact checks, model judges calibrated against your experts, and regression tests that run before every prompt, model or data change reaches production.
- Public leaderboards say little about your workflow; your own test cases do.
- Model judges must be checked against human labels before you trust their scores.
- Small test sets give noisy results, so report uncertainty, not just a percentage.
- Every prompt, model or data change reruns the suite before release.
What LLM evaluation is
LLM evaluation is measuring whether a system built on a language model does its specific job well enough, using fixed test cases, clear scoring rules and runs you can repeat. It has four parts: a dataset of realistic inputs with known good outcomes, scorers that judge each output, a runner that executes the whole set against any version of the system, and a report that shows what changed.
It is not reading a model leaderboard. Leaderboards rank models on general tasks. Your question is narrower: does this system, with this prompt, these documents and these tools, get your work right?
Who needs it
- Teams moving a prototype into production. If nobody can say how often it is wrong, it is not ready.
- Anyone changing models. Providers ship and retire models constantly; Anthropic retires Claude Sonnet 4.5 from its API on 30 November 2026 (Anthropic). Without an evaluation, every migration is a guess. Our AI model selection service uses the same suites.
- Systems whose output reaches customers, money or decisions about people, where an unmeasured error rate is a risk you cannot report on.
- Teams after an incident who need to prove the fix held and the problem cannot quietly return.
What we deliver
- Evaluation dataset
- Real cases from your documents, tickets or records, labelled with the right outcome by the people who do the work, grouped by category.
- Scoring rules
- A written definition of correct for every output type, agreed with your operators before any scoring code exists.
- Automated scorers
- Exact and normalised matches, schema validation, citation checks and rubric-based model judges, each matched to the output it scores.
- Judge calibration
- How often each model judge agrees with your experts, and where it does not.
- Regression suite
- The whole set wired into your release process, so a change that breaks a category is blocked.
- Failure taxonomy
- Errors sorted into causes (retrieval, reasoning, formatting, policy) so fixes target the right layer.
How outputs are scored
Different outputs need different scorers. Using a model judge where an exact check would do adds cost and noise.
| Output type | Scorer | What counts as a pass |
|---|---|---|
| Extracted fields (dates, amounts, GST numbers) | Exact or normalised match | Value equals the labelled value |
| Classification (ticket type, document type) | Accuracy per class | Correct label, reported per category |
| Grounded answers | Citation check plus faithfulness judge | Every claim traces to a cited passage |
| Drafted text (emails, summaries) | Rubric judge calibrated on expert labels | Meets each rubric point your team set |
| Agent tasks | End-state check plus step audit | Correct final state, no forbidden actions |
| Every output | Cost and latency | Within the budget per task |
Systems that search your documents get a second layer: we score the retrieval step on its own, asking whether the right passage came back at all, before judging the answer built on it. When an answer is wrong, that split tells you whether to fix the search or the prompt. Our guide to RAG retrieval evaluation covers the metrics.
A regression example
Illustrative example: a support-drafting system has a 240-case test set across six ticket categories. The team trials a newer, cheaper model. The overall pass rate rises from 86% to 89%, which looks like an easy upgrade. The per-category report shows the refund-policy category fell from 23 of 25 cases to 18 of 25, because the new model paraphrased a policy clause it used to quote. The release is blocked for that category, the prompt is changed to require quoting policy text, and the rerun shows 24 of 25.
Two lessons sit in that example. Overall averages hide category failures, so we always report by category. And small numbers are noisy: at an 85% pass rate on 100 cases, the standard error is about 3.6 percentage points, so a 3-point change can be chance. We show intervals, not just percentages.
How the work runs
- Discover: define good. We sit with the people who do the work, collect real cases including the awkward ones, and write down what a correct output looks like. Disagreements between experts surface here, and resolving them is often the most valuable part.
- Prove: build and calibrate. We build the scorers, calibrate any model judges against expert labels, and run a baseline on your current system so every later change has something to beat.
- Scale: gate releases and grow the set. The suite runs on every change. Weekly samples of live traffic are labelled and added, so the test set keeps pace with what users actually send. See our guide to versioning prompts and evaluations.
What it costs to run
The expensive part is expert time: labelling cases and agreeing on what good means. Running the suite is cheap. As a worked example at Anthropic’s published API prices on 19 October 2026 (Claude Sonnet 5.5 at US$2 per million input tokens and US$10 per million output tokens; Anthropic), judging 240 cases at 3,000 input and 300 output tokens each uses 720,000 input and 72,000 output tokens: US$1.44 plus US$0.72, about US$2.16 per full run before the system’s own model costs. Cost drivers are the number of categories, how specialised the labelling is, the judge model chosen and how often the suite runs.
Where evaluation goes wrong
- Judges that agree with themselves. A model grading output from its own family can share its blind spots. Calibrate, and use a different model as judge where you can.
- Teaching to the test. Prompts tuned until a small test set passes rarely generalise. Hold back a portion of cases that no one tunes against.
- Easy test sets. If the set only holds clean examples, it measures the demo. Production failures belong in the set within a week.
- Measuring the wrong thing. Fluency is easy to score and rarely the problem. Score the outcome the business cares about.
- Changing two things at once. A new model and a new prompt in the same release make any regression impossible to attribute. Change one variable per run.
Work this draws on
- GTM-Bench: a 72-task benchmark for AI agents doing go-to-market research, published by Blackpearl Group in June 2026 and co-authored by Levy Street team members. Weak leads score negatively, so an agent cannot win by flooding its output, and in the paper several general-purpose agents scored below zero overall. Our guide to AI agent evaluations explains the design choices.
- LucentBench: a forward-looking earnings-forecast benchmark described in the Lucent whitepaper, built to test financial research outputs against what actually happened.
Sources
- GTM-Bench: Evaluating Agentic AI Systems for Buyer/Seller Coherence in Go-to-Market Workflows, Blackpearl Group, 2026-06
- Pricing (Claude API), Anthropic, accessed 2026-10-19
- Model deprecations, Anthropic, accessed 2026-10-19
- Artificial intelligence (AI topic page), Office of the Privacy Commissioner, 2023-09-21
Written by the Levy Street team in Wellington. Last updated 19 October 2026.