AI quality review checklist.

Use this checklist whenever AI output is about to leave the reviewer’s hands: sent to a customer, filed, used in a decision or published. The Privacy Commissioner warns that generative AI tools “often produce very confident errors of fact or logic” (OPC). Confident errors are the ones a quick skim misses.

Choose the review level

The checklist

1. Facts and sources

• Every factual claim can be traced to the source material or a reliable reference.
• Every citation, case, link or document reference exists and says what the output claims.
• Nothing presented as current is out of date.
• The output does not answer beyond what the sources support.

2. Numbers and dates

• Figures match the source exactly, including units and currency.
• Any calculation has been re-done by hand or in a spreadsheet.
• GST treatment is stated correctly: inclusive or exclusive.
• Dates are correct and in New Zealand format, such as 20 October 2026.

3. People and privacy

• Names, roles and organisations are correct and spelt correctly, including macrons in Māori names.
• No personal information appears that the recipient should not see.
• Nothing about a person is guessed or inferred and presented as fact.
• Any decision about a person has been made by a person.

4. Completeness and instructions

• It does what was asked, in the format asked for.
• Exceptions, caveats and conditions in the source are carried through.
• Nothing important from the source is missing.
• Placeholders such as [name] or leftover instructions have been removed.

5. Tone and New Zealand context

• New Zealand law, regulators and terms are used, not US, UK or Australian ones.
• Spelling is New Zealand English.
• Tone suits the recipient and does not over-promise.
• Claims about products or services can be backed up.

6. Sign-off

• The reviewer would be comfortable putting their name to it.
• Level 3 outputs have a second reviewer’s sign-off.
• Any defect found is entered in the defect log.
• If the output was already used and wrong, it is reported as an incident.

Sampling rule for high-volume tasks

For tasks done many times a week, review 100% of outputs for the first two weeks. Move to sampling, for example one in five, only when the defect rate has stayed below the limit you set with the workflow owner. Return to 100% review for two weeks whenever the model, prompt or source documents change. Level 3 outputs are never sampled. Our guide to human review queues shows how to build this into a system.

Defect log template

Review the log monthly. Three defects of the same type mean the cause is upstream: a prompt, a missing or stale source document, or a step in the process.

Filled-in example

Illustrative example: a reviewer at a 45-person Tauranga insurance brokerage checks an AI-drafted letter explaining a policy change to a client, a Level 3 output under the broker client communication drafting pattern.

Four defects went into the log. Two had the same cause: the prompt was given last year’s policy wording. The team linked the prompt to the current wording, and the next 20 letters had no number or clause defects.

How to introduce the checklist

1. Set the level for each task type. Agree with the workflow owner which tasks are Level 1, 2 or 3.
2. Print or pin the checklist. Reviewers should not work from memory.
3. Start the defect log on day one. Even a short log reveals patterns within weeks.
4. Review the log monthly. Fix causes in prompts, sources or process.
5. Train reviewers on real examples. Use outputs with known defects; see checking AI answers against sources.

Common mistakes

• Skimming. Confident, fluent text hides errors. Read in full.
• Checking citations exist but not what they say. The New Zealand Law Society has warned that NZ courts are already dealing with references to cases that do not exist (NZLS).
• Assuming the AI is responsible. The Real Estate Authority says an error made by a generative AI tool is no defence to a breach of a licensee’s obligations (REA).
• Fixing the output but not the cause. Without the defect log, the same error returns next week.

Next step

Plan who reviews what, and when, with planning human review for AI output. When a defect reaches someone outside the business, record it in the AI incident log. For systems at volume, our LLM evaluation services automate much of this checking.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.