Checking AI answers against sources.

Owners, office managers, advisers and team leads who rely on AI answers need one habit above all: before an answer leaves your desk, split it into claims and check the ones that matter against sources you opened yourself. Building that habit is the goal of this lesson. You need 25 minutes, an AI assistant with web search switched on if your tool has it, and a factual question from your own work.

What AI hallucinations look like in business

A hallucination is a confident statement that is false or unsupported. Language models generate likely text rather than looking facts up, so errors arrive in the same fluent tone as correct answers. The Privacy Commissioner warns that these tools “often produce very confident errors of fact or logic” (OPC), and the New Zealand Law Society reported in December 2025 that New Zealand courts are already dealing with references to cases that do not exist (NZLS).

The four patterns you will meet most:

The claim table method

Verification works claim by claim, because one answer often mixes correct and wrong statements. Put each claim in a row: the claim, the source given, what the source actually says, and a verdict of supported, partly supported, unsupported or wrong.

Check in proportion to the stakes. Anything you will publish, send to a customer, or act on with money, law, safety or someone’s personal information must trace to a primary source you opened: legislation.govt.nz, a regulator such as the Privacy Commissioner or Inland Revenue, or the vendor’s own documentation. Blogs and AI summaries are leads, not evidence. If you cannot verify a claim in reasonable time, leave it out.

Exercise: build a claim table

1. Ask a real question. Choose a factual question from your work with consequences, such as a compliance rule, a supplier’s terms or a product specification. Leave out personal details.
2. Ask for claims with sources. Use the prompt in the worked example below so the answer comes back as numbered statements, each with a source link.
3. Rank by stakes. Mark which claims would cost money, breach a rule or mislead a customer if wrong. Check those first.
4. Open every source yourself. Find the exact sentence. Check the date and that it is a New Zealand source where New Zealand law applies.
5. Give each claim a verdict. Supported, partly supported, unsupported or wrong, with a one-line note.
6. Rebuild the answer. Write the final version using only supported claims, with your own source links, and note what you could not confirm.

Worked example: a vet clinic’s misdirected email

Illustrative example: the office manager at a Palmerston North vet clinic discovers that an invoice listing one client’s name, address and pet’s treatment was emailed to a different client. She asks an AI assistant what the clinic must do.

An email containing one client's name, address and their pet's treatment details
was sent to the wrong client at our veterinary clinic in New Zealand.

List, as numbered claims, what New Zealand law requires us to do.
For each claim give the specific source (Act and section, or official web page)
and a link. If you are not sure a claim applies in New Zealand, say so.
Do not give a combined summary; claims only.

Example output (illustrative and abridged), with her verdicts added after checking:

The second claim is a classic hallucination pattern: a hard 72-hour deadline appears in some overseas laws, such as the EU’s GDPR, and the model presented it as New Zealand law. Her rebuilt answer says: assess whether the breach is likely to cause serious harm using OPC’s self-assessment tool; if it is notifiable, notify OPC as soon as practicable (ideally within 72 hours) and tell the affected client; record the decision either way. Whether this particular email is notifiable is her judgement, not the AI’s. The Privacy Act 2020 is the primary source for every row.

Where checking goes wrong

• Stopping at the link. A real, official-looking URL is not evidence until you find the supporting sentence.
• Asking the AI to verify itself. It may confirm a wrong claim or abandon a right one. Use it to list claims, not to judge them.
• Checking the easy claims. People verify the obvious points and skip the one with consequences. Rank by stakes first.
• Trusting old pages. Check the date on the source. Rules, prices and product features change.
• Assuming grounded tools are immune. Tools that answer from your uploaded documents, such as Gemini Notebook (formerly NotebookLM), reduce invention but can still misread a document. The lesson on summarising documents with citations covers checking those.

Systems built for high-stakes work should make checking easy by design: Levy Street’s own research platform, Lucent, returns outputs with citations a reviewer can follow back to the source. For teams reviewing AI output at volume, our AI quality review checklist and LLM evaluation services take the same idea further. If the AI’s mistake has already reached a customer, see managing AI mistakes and escalation.

Check your work

• I asked for the answer as numbered claims with sources.
• I ranked the claims by what would happen if they were wrong.
• I opened every source for the high-stakes claims and found the supporting sentence.
• I checked that New Zealand sources were used where New Zealand rules apply.
• Every claim has a verdict, and wrong or unsupported claims are removed.
• My final answer cites sources I opened myself.

Next lesson: Giving AI useful context shows how to supply your own facts so there is less to check in the first place.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.