Planning human review for AI output.

Goal

Write a human review plan that matches the level of checking to the risk of each AI use, and keeps reviewers genuinely checking.

Who it is for

Managers, team leads and system owners in New Zealand organisations where AI output reaches customers, candidates, clients or decisions.

Time

30 minutes.

You need

ChatGPT, Claude, Copilot or Gemini, the fictional list of AI uses below, and later a list of your own team’s AI uses.

You will produce

A review plan table with tiers, reviewers, methods, sample rates and checklist items.

What is a human review plan?

A human review plan decides, before AI output is used, which outputs a person checks, who checks them, against what, how many, and what happens when they find a problem. Without one, review drifts to whoever has time, and stops entirely when everyone is busy.

New Zealand guidance is consistent on the principle. The Privacy Commissioner’s guidance on AI and the privacy principles says to “ensure human review prior to acting on AI outputs to reduce risks of inaccuracy and bias.” MBIE’s quickstart for businesses says a human-in-the-loop “is needed where important decisions and actions are being made or informed by AI.”

How much review does each output need?

Ask three questions of each AI use: how much harm a mistake could do (impact), whether it can be corrected before harm occurs (reversibility), and whether it leaves the business (visibility). Then assign a tier.

Where should review sit in the workflow?

Put the check where an error becomes irreversible, usually just before something leaves the business or a decision takes effect. And make sure the reviewer sees what the AI excluded as well as what it produced.

MBIE’s responsible AI guidance includes a hypothetical construction firm, BigBuild, that trials CV screening with “a human reviewer for all shortlisted applications.” Despite equal numbers of women applying, only 3 of the 10 shortlisted were women, and rejected applicants’ CVs proved at least as strong. The reviewer was real; the review was in the wrong place. Screening decisions can also engage the Human Rights Act 1993, which prohibits discrimination on grounds including sex and age.

Exercise: tier a recruitment agency’s AI uses

The agency below is fictional.

Illustrative AI uses. Fictional recruitment agency, Wellington,
12 consultants.

U1 Drafting job ads from client briefs
U2 Ranking applicants' CVs against the job criteria
U3 Summarising interview notes into reports sent to clients
U4 Drafting rejection emails to unsuccessful candidates
U5 Summarising reference check calls
U6 Writing LinkedIn posts about the agency
U7 Tidying internal team meeting notes

1. Run the planning prompt in the worked example below.
2. Challenge the tiers. Check that U2, U3 and U5 are high, since each shapes a decision about someone’s employment. Move anything you disagree with and note why.
3. Check placement. For U2, confirm the plan has a person reviewing applicants the AI ranked low, not only the shortlist.
4. Name reviewers. Replace roles with real people in your own version, and check they have time budgeted.
5. Add a reviewer test. Choose one medium or high use and plan to seed a known error once a month to see whether it is caught.
6. Decide what gets logged. Record review outcomes, especially overrides, so you can see whether the AI or the review is weakening.

Worked example: the review planning prompt

For each AI use below, produce a table with columns:
Use | Impact | Reversible? | Leaves the business? | Tier (low /
medium / high / not for AI) | Review point | Reviewer role |
Method and sample rate | Three checklist items.

Rules: anything that shapes a decision about a person's job,
money, safety or rights is high. Reviews must happen before an
error becomes irreversible. For screening or ranking, the review
must cover items the AI ranked low or excluded.

[paste AI uses]

Example output (illustrative and abridged; your tool’s wording will differ):

U2 Ranking CVs | High | No, once candidates are rejected | No |
High | Before any candidate is rejected | Senior consultant |
Every application reviewed; AI ranking shown as a suggestion |
- Read all applications ranked below the cut-off
- Compare shortlist mix with applicant pool
- Record reasons for each rejection in your own words

U4 Rejection emails | Medium | No, once sent | Yes | Medium |
Before sending | Consultant who ran the role | Every email |
- Name and role correct
- No reasons given that were not part of the decision
- Tone respectful

The useful detail is in U2’s checklist. Comparing the shortlist with the whole applicant pool is what would have caught BigBuild’s problem.

How do you keep reviewers alert?

Review quality decays as AI improves, because reviewers see so many good outputs that they stop looking. OWASP’s Top 10 for LLM Applications 2026 warns that “approval fatigue degrades reviewer judgment at volume.” Keep volumes manageable, show the source next to the output, use checklists that force specific checks, and track the override rate. If reviewers never change anything, find out whether the AI is that good or the review has become a habit.

Where this goes wrong

• Review of the output, not the omissions. Filters and rankings hide their errors in what they leave out.
• No time budgeted. Review squeezed into a busy day becomes a glance.
• The wrong reviewer. Someone without the expertise or authority to reject the output adds little. Our Property Management Agent for APL Property is built so staff keep the decisions.
• No record. Without logged outcomes you cannot show review happened or learn from what it caught. Log errors in your escalation process for AI mistakes.

Check your work

• Every AI use in my team has a tier based on impact, reversibility and visibility.
• Each review sits before the point where an error becomes irreversible.
• Screening or ranking uses include review of what the AI excluded.
• Every reviewer is named, qualified for the tier, and has time budgeted.
• I have a plan to seed known errors and track override rates.
• Review outcomes are logged somewhere the team can see.

Where to go next

This is the last lesson in this part of the pathway; return to the Levy Street learning hub to choose your next topic. For drafting with a review step built in, see drafting emails with human review, and for a recruitment workflow in detail, interview note summarisation. Use the AI quality review checklist as a starting point for reviewer checklists. Engineers building review into systems should read human review queues. For organisation-wide review standards, see our AI governance advisory.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.