# AI pilot scorecard.

URL: https://levystreet.com/resources/ai-pilot-scorecard/
Updated: 2026-10-19

Free AI pilot scorecard: eight weighted criteria scored 0 to 4, hard gates and a decision rule to scale, fix or stop an AI trial, with a retail example.

## Short answer

An AI pilot scorecard decides, with numbers agreed in advance, whether a trial of an AI system should scale, change or stop. This scorecard rates eight weighted criteria from 0 to 4, including accuracy on real cases, time saved after review, exceptions, adoption and cost per task. Five hard gates stop a pilot regardless of its score.

- Set criteria, weights and thresholds before the pilot starts, not after.
- Score accepted outputs on real cases, not drafts produced in a demo.
- Hard gates, such as an unreviewed output reaching a customer, override the score.
- 75 or more scales; 50 to under 75 is fixed and re-run; under 50 stops.

Fill in Part A and the thresholds in Part C on day one of the pilot, and have the sponsor sign them. MBIE's quickstart for businesses suggests starting small and trialling systems before committing ([MBIE](https://www.mbie.govt.nz/assets/businesses-using-ai-tools.pdf)); this scorecard makes the trial end in a decision.

## The AI pilot scorecard

### Part A: Pilot details

| Field | Entry |
|---|---|
| Workflow and system | |
| Pilot dates and review date | |
| Users and teams | |
| Baseline period and method | |
| Cases processed in the pilot | |
| Sample size checked for accuracy | |
| Scored by | |

### Part B: Hard gates

Any "no" means stop or redesign, whatever the score.


- No AI output that required review reached a customer or third party unreviewed.
- No privacy breach occurred that was likely to cause serious harm.
- Errors on high-impact cases stayed at or below the agreed limit.
- Users could escalate or switch the system off, and that worked when used.
- A baseline was measured before the pilot started.


### Part C: Scored criteria

Score each criterion 0 to 4. A result at an anchor gets that score; a result between two anchors gets the score in between (1 or 3). The anchors below are examples: change them to suit your workflow before the pilot starts.

| Criterion | Weight | How measured | 0 | 2 | 4 |
|---|---|---|---|---|---|
| 1. Accuracy on real cases | 25 | Share of sampled outputs accepted with no or minor edits | Under 70% | 85% | 95% or more |
| 2. Time per task after review | 20 | Reduction against baseline, including review | None | 30% less | 60% less |
| 3. Exception rate | 10 | Share of cases the AI could not handle | Over 40% | 20% | Under 10% |
| 4. Adoption | 15 | Share of eligible tasks run through the system, final two weeks | Under 30% | 60% | 85% or more |
| 5. User confidence | 10 | Average agreement, 1 to 5: "After my usual check, I trust the output" | Under 2.5 | 3.5 | 4.5 or more |
| 6. Cost per task | 10 | Actual against budgeted cost per task | Over 150% | 100% | Under 75% |
| 7. Reliability | 5 | Share of runs that failed or timed out | Over 5% | 2% | Under 0.5% |
| 8. Controls working | 5 | Logs, review queue and audit sample | Missing | Present with gaps | All evidenced |

**Weighted score** = the sum of (score ÷ 4 × weight). The maximum is 100.

### Part D: Decision rule

| Result | Decision |
|---|---|
| All gates pass and score 75 or more | Scale: plan production, monitoring and support |
| All gates pass and score 50 to under 75 | Fix and re-run: list the named fixes, re-score after four weeks |
| Any gate fails, or score under 50 | Stop or redesign: record what was learned |

## How to run the pilot with the scorecard


1. **Measure the baseline first.** Time 20 or more real tasks before the AI is switched on. Our [AI ROI calculator](/resources/ai-roi-calculator/) explains which numbers matter.
2. **Agree anchors and sign them.** The sponsor and workflow owner sign Part C before day one.
3. **Sample outputs every week.** Check at least 20 a week against the [AI quality review checklist](/resources/ai-quality-review-checklist/) and record accepted, edited or rejected.
4. **Survey users at the end.** Use the confidence questions from the [AI adoption survey](/resources/ai-adoption-survey/).
5. **Score, then decide in one meeting.** Two people score independently, compare, and apply the decision rule.


## Worked example

Illustrative example: a six-week pilot of AI-drafted replies to customer questions at a 20-person Auckland online homeware retailer. 1,140 enquiries went through the system, and 240 drafts were sampled. All five gates passed.

| Criterion | Result | Score | Weighted |
|---|---|---|---|
| 1. Accuracy | 88% accepted with minor edits | 3 | 18.75 |
| 2. Time per task | 7.5 to 4.1 minutes, 45% less | 3 | 15.0 |
| 3. Exceptions | 14% | 3 | 7.5 |
| 4. Adoption | 72% of eligible enquiries | 3 | 11.25 |
| 5. Confidence | Average 3.9 | 3 | 7.5 |
| 6. Cost per task | 80% of budget | 3 | 7.5 |
| 7. Reliability | 1.1% failed runs | 3 | 3.75 |
| 8. Controls | Review queue and logs; sampling missed in week 2 | 2 | 2.5 |
| **Total** | | | **73.75** |

Decision: fix and re-run, just under the line for scaling. Most rejected drafts were about returns, because the returns policy in the knowledge base was out of date. The team updated it, added weekly sampling to a calendar, and re-scored after four weeks. In [customer question answering](/industries/retail/customer-question-answering/) projects, accuracy depends on the source documents as much as on the model.

## Common mistakes

- **Setting thresholds after seeing results.** The scorecard then confirms whatever people hoped.
- **Counting drafts, not accepted outputs.** A draft rewritten from scratch is a failure, not a time saving.
- **Leaving out review time.** The Privacy Commissioner expects human review before acting on AI output ([OPC](https://www.privacy.org.nz/resources-and-learning/a-z-topics/ai/generative-artificial-intelligence/)); that time is part of the cost of every task.
- **One enthusiastic user.** Adoption by the champion alone does not predict adoption by the team.
- **Testing on easy cases.** Pilot on the real mix, including the messy ones.

## Next step

If the pilot scales, plan monitoring before go-live; our lesson on [evaluating an AI pilot](/learn/evaluating-an-ai-pilot/) covers the handover, and [AI agent evaluations](/engineering/ai-agent-evaluations/) explains how to keep testing once it is live. If you need a partner to run a time-boxed trial, see our [AI proof of concept](/services/ai-proof-of-concept/) service.

## Questions people ask

### How do you evaluate an AI pilot?

Agree before the pilot starts what you will measure, how, and what result means scale, fix or stop. Measure a baseline first. During the pilot, sample real outputs and record whether they were accepted, how long review took, how many cases the AI could not handle, how many eligible tasks went through it, and what it cost. Score at the end against the thresholds you set.

### How long should an AI pilot run?

Long enough to cover a normal cycle of the work and get past the first week of learning, which for most office workflows means four to eight weeks. Shorter pilots tend to measure novelty and setup problems. Longer ones drift without a decision. Put the review date in the business case and keep it.

### What accuracy should an AI pilot reach?

It depends on what an error costs and whether a person reviews every output. Drafts that a person always checks can be useful at 80 to 90% accepted with minor edits. Anything that acts without review needs a much higher bar on high-impact cases. Set the threshold with the workflow owner before testing, and write it in the scorecard.

### Who should score the pilot?

The workflow owner, with evidence from the people who used it and whoever sampled outputs. The person who built or sold the system should supply data but not set the score. Two people scoring independently and then agreeing gives a fairer result than one enthusiastic sponsor.


## Sources

- [Quickstart: Businesses using AI tools](https://www.mbie.govt.nz/assets/businesses-using-ai-tools.pdf)
- [Generative Artificial Intelligence](https://www.privacy.org.nz/resources-and-learning/a-z-topics/ai/generative-artificial-intelligence/)
- [Privacy Act 2020](https://www.legislation.govt.nz/act/public/2020/31/en/latest/)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.