Evaluating an AI pilot.

Every AI trial needs an ending: a recommendation to keep it, change it or stop it. For the operations manager, practice manager or project lead who owns that trial, the aim here is a decision based on evidence: a small test set from your own past work, scored against answers written before the AI saw it, and judged by criteria agreed in advance. Allow 45 minutes to design the evaluation, plus time to run the cases.

Why AI pilots need a test set

Without a test set, pilots are judged on impressions. The person who championed the pilot remembers the good outputs; the sceptic remembers the one that went badly wrong. Neither is evidence.

A test set fixes this. It is a collection of real past cases with known correct answers, used to check the AI before it touches live work, and re-used after every change. It also gives you a reference when the vendor updates the model and you need to know whether anything got worse.

Pass criteria come first

Write the pass criteria before running a single case, and get the decision-maker to agree to them. Otherwise the bar moves to fit the result.

Then the decision rule: Go if every threshold is met; Adjust if any threshold is missed and the failures share a fixable cause, then re-run the full set; Stop if critical errors persist after two rounds of changes.

Exercise: build and run a pilot evaluation

1. Define the job. One sentence on what the AI does, plus the exact output fields, such as category, urgency and draft reply.
2. Collect 20 to 30 past cases. Use real ones, de-identified. About a third should be awkward: ambiguous, two issues in one message, angry, or missing key details.
3. Write the answer key first. The person who normally does the work writes the correct output for each case before anyone runs the AI.
4. Agree pass criteria and the decision rule. Use the table above as a starting point and sign it off with the decision-maker.
5. Run every case the same way. Same prompt, same tool, each case in a fresh chat or a single consistent batch. Save every output.
6. Score and decide. Mark each output against the key, tally the criteria, and apply the decision rule as written. After any change, re-run all cases, not just the failures.

Worked example: maintenance request triage in Lower Hutt

Illustrative example: a property management firm in Lower Hutt receives 60 to 80 tenant maintenance requests a week by email. It pilots AI to suggest an urgency level and trade for each, with a property manager confirming every one.

You triage tenant maintenance requests for a New Zealand property manager.
For the request below, return exactly these lines:

URGENCY: Urgent / Soon / Routine
TRADE: Plumber / Electrician / Builder / Locksmith / Appliance / Other
REASON: one sentence quoting the words that drove your decision
UNCLEAR: anything missing that a property manager should ask about, or "none"

Rules: if the request mentions water near electrics, a gas smell, no power,
no water, or a door or window that will not lock, URGENCY is Urgent.
If you are unsure between two urgency levels, choose the higher one.

REQUEST:
[paste de-identified request]

Example output for one case (illustrative):

URGENCY: Urgent
TRADE: Electrician
REASON: "water dripping out of the hallway light" indicates water near electrics.
UNCLEAR: whether power to the hallway circuit has been turned off.

The first run used a prompt without the Rules section. Results across 30 cases:

The first run missed the accuracy threshold and had one critical error, both caused by the same missing rule, so the decision was Adjust. After the second run met every threshold, the firm moved to supervised live use, with a property manager confirming every suggestion for four weeks before reviewing the arrangement. Our maintenance request triage guide covers the full workflow, and the property management agent we built with APL Property shows the same principle in production: the system does the administrative work and staff keep the decisions.

Where pilot evaluations go wrong

• An easy test set. If every case is typical, the pilot passes and live work fails on the first odd email. Include the awkward third.
• Answer keys written after the fact. Seeing the AI’s answer first biases the key toward it.
• One run. Outputs vary between runs. Run the set twice and treat inconsistent cases as failures.
• Tuning to the test. Fixing prompts until these 30 cases pass can make the AI worse at the next 30. Keep a few fresh cases aside for a final check.
• Trusting the reviewer completely. The Privacy Commissioner notes that people overseeing automated systems can find it hard to notice errors, a problem called automation blindness (OPC). Spot-check reviewed outputs, too.
• No exit date. A pilot without an end date becomes a permanent half-system. Set the decision date at the start.

The AI pilot scorecard gives you this structure ready to fill in. For pilots that need a production-grade build to test properly, our AI proof of concept service is designed around exactly this kind of evaluation, and our engineering guide to AI agent evaluations covers automated testing at larger scale.

Check your evaluation

• The job and output fields are defined in writing.
• The test set has 20 to 30 real, de-identified cases, about a third of them awkward.
• The answer key was written before any AI output was seen.
• Pass criteria and the decision rule were agreed before the first run.
• Critical errors are defined and counted separately.
• I re-ran the full set after every change and recorded each run.

Next lesson: Choosing business AI subscriptions covers which tools and plans to pay for once a pilot passes.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.