Short answer
An AI pilot scorecard decides, with numbers agreed in advance, whether a trial of an AI system should scale, change or stop. This scorecard rates eight weighted criteria from 0 to 4, including accuracy on real cases, time saved after review, exceptions, adoption and cost per task. Five hard gates stop a pilot regardless of its score.
- Set criteria, weights and thresholds before the pilot starts, not after.
- Score accepted outputs on real cases, not drafts produced in a demo.
- Hard gates, such as an unreviewed output reaching a customer, override the score.
- 75 or more scales; 50 to under 75 is fixed and re-run; under 50 stops.
Fill in Part A and the thresholds in Part C on day one of the pilot, and have the sponsor sign them. MBIE’s quickstart for businesses suggests starting small and trialling systems before committing (MBIE); this scorecard makes the trial end in a decision.
The AI pilot scorecard
Part A: Pilot details
| Field | Entry |
|---|---|
| Workflow and system | |
| Pilot dates and review date | |
| Users and teams | |
| Baseline period and method | |
| Cases processed in the pilot | |
| Sample size checked for accuracy | |
| Scored by |
Part B: Hard gates
Any “no” means stop or redesign, whatever the score.
- No AI output that required review reached a customer or third party unreviewed.
- No privacy breach occurred that was likely to cause serious harm.
- Errors on high-impact cases stayed at or below the agreed limit.
- Users could escalate or switch the system off, and that worked when used.
- A baseline was measured before the pilot started.
Part C: Scored criteria
Score each criterion 0 to 4. A result at an anchor gets that score; a result between two anchors gets the score in between (1 or 3). The anchors below are examples: change them to suit your workflow before the pilot starts.
| Criterion | Weight | How measured | 0 | 2 | 4 |
|---|---|---|---|---|---|
| 1. Accuracy on real cases | 25 | Share of sampled outputs accepted with no or minor edits | Under 70% | 85% | 95% or more |
| 2. Time per task after review | 20 | Reduction against baseline, including review | None | 30% less | 60% less |
| 3. Exception rate | 10 | Share of cases the AI could not handle | Over 40% | 20% | Under 10% |
| 4. Adoption | 15 | Share of eligible tasks run through the system, final two weeks | Under 30% | 60% | 85% or more |
| 5. User confidence | 10 | Average agreement, 1 to 5: “After my usual check, I trust the output” | Under 2.5 | 3.5 | 4.5 or more |
| 6. Cost per task | 10 | Actual against budgeted cost per task | Over 150% | 100% | Under 75% |
| 7. Reliability | 5 | Share of runs that failed or timed out | Over 5% | 2% | Under 0.5% |
| 8. Controls working | 5 | Logs, review queue and audit sample | Missing | Present with gaps | All evidenced |
Weighted score = the sum of (score ÷ 4 × weight). The maximum is 100.
Part D: Decision rule
| Result | Decision |
|---|---|
| All gates pass and score 75 or more | Scale: plan production, monitoring and support |
| All gates pass and score 50 to under 75 | Fix and re-run: list the named fixes, re-score after four weeks |
| Any gate fails, or score under 50 | Stop or redesign: record what was learned |
How to run the pilot with the scorecard
- Measure the baseline first. Time 20 or more real tasks before the AI is switched on. Our AI ROI calculator explains which numbers matter.
- Agree anchors and sign them. The sponsor and workflow owner sign Part C before day one.
- Sample outputs every week. Check at least 20 a week against the AI quality review checklist and record accepted, edited or rejected.
- Survey users at the end. Use the confidence questions from the AI adoption survey.
- Score, then decide in one meeting. Two people score independently, compare, and apply the decision rule.
Worked example
Illustrative example: a six-week pilot of AI-drafted replies to customer questions at a 20-person Auckland online homeware retailer. 1,140 enquiries went through the system, and 240 drafts were sampled. All five gates passed.
| Criterion | Result | Score | Weighted |
|---|---|---|---|
| 1. Accuracy | 88% accepted with minor edits | 3 | 18.75 |
| 2. Time per task | 7.5 to 4.1 minutes, 45% less | 3 | 15.0 |
| 3. Exceptions | 14% | 3 | 7.5 |
| 4. Adoption | 72% of eligible enquiries | 3 | 11.25 |
| 5. Confidence | Average 3.9 | 3 | 7.5 |
| 6. Cost per task | 80% of budget | 3 | 7.5 |
| 7. Reliability | 1.1% failed runs | 3 | 3.75 |
| 8. Controls | Review queue and logs; sampling missed in week 2 | 2 | 2.5 |
| Total | 73.75 |
Decision: fix and re-run, just under the line for scaling. Most rejected drafts were about returns, because the returns policy in the knowledge base was out of date. The team updated it, added weekly sampling to a calendar, and re-scored after four weeks. In customer question answering projects, accuracy depends on the source documents as much as on the model.
Common mistakes
- Setting thresholds after seeing results. The scorecard then confirms whatever people hoped.
- Counting drafts, not accepted outputs. A draft rewritten from scratch is a failure, not a time saving.
- Leaving out review time. The Privacy Commissioner expects human review before acting on AI output (OPC); that time is part of the cost of every task.
- One enthusiastic user. Adoption by the champion alone does not predict adoption by the team.
- Testing on easy cases. Pilot on the real mix, including the messy ones.
Next step
If the pilot scales, plan monitoring before go-live; our lesson on evaluating an AI pilot covers the handover, and AI agent evaluations explains how to keep testing once it is live. If you need a partner to run a time-boxed trial, see our AI proof of concept service.
Sources
- Quickstart: Businesses using AI tools, Ministry of Business, Innovation and Employment, accessed 19 October 2026
- Generative Artificial Intelligence, Office of the Privacy Commissioner, 15 June 2023
- Privacy Act 2020, New Zealand Legislation, version as at 1 October 2026
Written by the Levy Street team in Wellington. Last updated 19 October 2026.