# Evaluating an AI pilot.

URL: https://levystreet.com/learn/evaluating-an-ai-pilot/
Updated: 2026-10-19

AI evaluation for business pilots: build a test set, write pass criteria before you run it, score critical errors and decide go, adjust or stop.

## Short answer

AI evaluation for a business pilot means testing the AI against real past cases with known correct answers before trusting it with live work. Collect 20 to 30 cases including awkward ones, write the expected answer for each first, agree pass criteria in advance, including zero tolerance for critical errors, then score every output and decide to go, adjust and re-test, or stop.

- Write the answer key before you see any AI output.
- Agree pass criteria and the decision rule before running the test.
- Separate critical errors from minor ones; one critical miss can fail a pilot.
- Re-run the whole test set after every prompt change.
- A person reviewing outputs can miss errors too; plan for automation blindness.

Every AI trial needs an ending: a recommendation to keep it, change it or stop it. For the operations manager, practice manager or project lead who owns that trial, the aim here is a decision based on evidence: a small test set from your own past work, scored against answers written before the AI saw it, and judged by criteria agreed in advance. Allow 45 minutes to design the evaluation, plus time to run the cases.

## Why AI pilots need a test set

Without a test set, pilots are judged on impressions. The person who championed the pilot remembers the good outputs; the sceptic remembers the one that went badly wrong. Neither is evidence.

A test set fixes this. It is a collection of real past cases with known correct answers, used to check the AI before it touches live work, and re-used after every change. It also gives you a reference when the vendor updates the model and you need to know whether anything got worse.

## Pass criteria come first

Write the pass criteria before running a single case, and get the decision-maker to agree to them. Otherwise the bar moves to fit the result.

| Criterion | Example threshold | Why it matters |
|---|---|---|
| Correct category or decision | At least 27 of 30 (90%) | Measures the core job |
| Critical errors | Zero | One urgent case treated as routine can cause real harm |
| Draft usable without major edits | At least 21 of 30 (70%) | Measures whether it saves time |
| Review time per case | Under 2 minutes median | A slow review erases the saving |

Then the decision rule: **Go** if every threshold is met; **Adjust** if any threshold is missed and the failures share a fixable cause, then re-run the full set; **Stop** if critical errors persist after two rounds of changes.

## Exercise: build and run a pilot evaluation


1. **Define the job.** One sentence on what the AI does, plus the exact output fields, such as category, urgency and draft reply.
2. **Collect 20 to 30 past cases.** Use real ones, de-identified. About a third should be awkward: ambiguous, two issues in one message, angry, or missing key details.
3. **Write the answer key first.** The person who normally does the work writes the correct output for each case before anyone runs the AI.
4. **Agree pass criteria and the decision rule.** Use the table above as a starting point and sign it off with the decision-maker.
5. **Run every case the same way.** Same prompt, same tool, each case in a fresh chat or a single consistent batch. Save every output.
6. **Score and decide.** Mark each output against the key, tally the criteria, and apply the decision rule as written. After any change, re-run all cases, not just the failures.


## Worked example: maintenance request triage in Lower Hutt

Illustrative example: a property management firm in Lower Hutt receives 60 to 80 tenant maintenance requests a week by email. It pilots AI to suggest an urgency level and trade for each, with a property manager confirming every one.

```text
You triage tenant maintenance requests for a New Zealand property manager.
For the request below, return exactly these lines:

URGENCY: Urgent / Soon / Routine
TRADE: Plumber / Electrician / Builder / Locksmith / Appliance / Other
REASON: one sentence quoting the words that drove your decision
UNCLEAR: anything missing that a property manager should ask about, or "none"

Rules: if the request mentions water near electrics, a gas smell, no power,
no water, or a door or window that will not lock, URGENCY is Urgent.
If you are unsure between two urgency levels, choose the higher one.

REQUEST:
[paste de-identified request]
```

Example output for one case (illustrative):

```text
URGENCY: Urgent
TRADE: Electrician
REASON: "water dripping out of the hallway light" indicates water near electrics.
UNCLEAR: whether power to the hallway circuit has been turned off.
```

The first run used a prompt without the Rules section. Results across 30 cases:

| Criterion | First run | After adding rules |
|---|---|---|
| Correct urgency and trade | 26 of 30 | 29 of 30 |
| Critical errors | 1 (light fitting leak marked Routine, Plumber) | 0 |
| Usable without major edits | 23 of 30 | 25 of 30 |
| Median review time | 1.5 minutes | 1.5 minutes |

The first run missed the accuracy threshold and had one critical error, both caused by the same missing rule, so the decision was Adjust. After the second run met every threshold, the firm moved to supervised live use, with a property manager confirming every suggestion for four weeks before reviewing the arrangement. Our [maintenance request triage](/industries/property-management/maintenance-request-triage/) guide covers the full workflow, and the [property management agent](/portfolio/property-management-agent/) we built with APL Property shows the same principle in production: the system does the administrative work and staff keep the decisions.

## Where pilot evaluations go wrong

- **An easy test set.** If every case is typical, the pilot passes and live work fails on the first odd email. Include the awkward third.
- **Answer keys written after the fact.** Seeing the AI's answer first biases the key toward it.
- **One run.** Outputs vary between runs. Run the set twice and treat inconsistent cases as failures.
- **Tuning to the test.** Fixing prompts until these 30 cases pass can make the AI worse at the next 30. Keep a few fresh cases aside for a final check.
- **Trusting the reviewer completely.** The Privacy Commissioner notes that people overseeing automated systems can find it hard to notice errors, a problem called automation blindness ([OPC](https://www.privacy.org.nz/assets/New-order/Resources-/Publications/Guidance-resources/AI-Guidance-Resources-/AI-and-the-Information-Privacy-Principles.pdf)). Spot-check reviewed outputs, too.
- **No exit date.** A pilot without an end date becomes a permanent half-system. Set the decision date at the start.

The [AI pilot scorecard](/resources/ai-pilot-scorecard/) gives you this structure ready to fill in. For pilots that need a production-grade build to test properly, our [AI proof of concept](/services/ai-proof-of-concept/) service is designed around exactly this kind of evaluation, and our engineering guide to [AI agent evaluations](/engineering/ai-agent-evaluations/) covers automated testing at larger scale.

## Check your evaluation


- The job and output fields are defined in writing.
- The test set has 20 to 30 real, de-identified cases, about a third of them awkward.
- The answer key was written before any AI output was seen.
- Pass criteria and the decision rule were agreed before the first run.
- Critical errors are defined and counted separately.
- I re-ran the full set after every change and recorded each run.


Next lesson: [Choosing business AI subscriptions](/learn/choosing-business-ai-subscriptions/) covers which tools and plans to pay for once a pilot passes.

## Questions people ask

### How do you evaluate an AI pilot?

Define the job and the output you expect, collect 20 to 30 real past cases, and have the person who normally does the work write the correct answer for each. Agree pass criteria in advance, such as category accuracy and zero critical misses. Run every case through the AI with the same prompt, score each output against the answer key, and apply the decision rule you agreed.

### How many test cases does an AI pilot need?

For a small business workflow, 20 to 30 well-chosen cases is enough to make a decision, provided roughly a third are edge cases: ambiguous requests, two issues in one message, unusual wording or missing information. A larger set gives more confidence and is worth building if the workflow will run thousands of times a year or the cost of errors is high.

### What accuracy is good enough for an AI pilot?

It depends on what happens when the AI is wrong and who catches it. For drafts a person always reviews, 85 to 90 percent acceptable outputs may be fine. For anything where an error reaches a customer or delays something urgent, the bar is zero critical errors on the test set. Set the threshold before testing, based on the cost of a mistake.

### What should we do if an AI pilot fails?

Look at which cases failed and why. If the failures share a cause, such as a missing rule or missing context, fix it and re-run the entire test set. If critical errors persist after two rounds of changes, stop. A stopped pilot that cost a few weeks is a good result compared with a weak system running live. Record what you learned for the next candidate workflow.


## Sources

- [Artificial intelligence and the Information Privacy Principles](https://www.privacy.org.nz/assets/New-order/Resources-/Publications/Guidance-resources/AI-Guidance-Resources-/AI-and-the-Information-Privacy-Principles.pdf)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.