Writing AI vendor requirements.

Goal: Turn a page of rough notes into numbered, testable requirements for an AI product, each with the evidence a vendor must supply. Who it is for: Owners, operations and IT leads, and anyone running an AI purchase without a procurement team. Time: 45 minutes, plus your notes on what the tool must do and, ideally, the budget from the previous lesson.

Three AI vendors will give you three excellent demos, each on data chosen to show its product at its best, each answering the questions it prefers. Requirements reverse that. They make every vendor answer the same questions, about your work, with evidence you can check. MBIE’s Responsible AI guidance for businesses (July 2025) recommends exactly this: be clear on business needs, document evaluation criteria, and “consider requesting a trial (isolated from other technical systems)”.

What makes a requirement testable

A testable requirement says what must be true, how you will check it, and what evidence the vendor provides. Most weak requirements are adjectives.

The New Zealand data-handling requirements

Data-handling requirements decide which privacy situation you are in. The Privacy Commissioner’s guidance on working with third-party providers explains that when a provider stores or processes personal information solely on your behalf, you remain fully responsible for it. If the provider also uses it for its own purposes (the guidance gives AI training data as an example), both of you hold the information, sharing it may be a disclosure, and IPP12 may apply if the provider is outside New Zealand. The same guidance recommends a plan for who notifies the Commissioner and affected people after a breach.

So the core requirements are: no training or other own-purpose use of your data; disclosed storage and processing locations; a sub-processor list with notice of changes; breach notification to you within a set period; and deletion with confirmation when you leave. This is general information, not legal advice.

Exercise: turn rough notes into vendor requirements

1. Write the problem and the measure. One paragraph: the job, the volume, and the number that should improve.
2. List rough needs. Write everything you care about as bullet points, without polishing. Mark each must-have or nice-to-have.
3. Convert with AI. Paste your notes into ChatGPT, Claude, Copilot or Gemini with the worked example prompt. Ask it to flag anything too vague to test.
4. Add the evidence column. For each requirement, confirm the evidence is something a vendor can actually supply: a document, a demonstration, a contract clause or a trial result.
5. Check every legal reference yourself. Remove any regulatory requirement the AI added that you cannot confirm; check the rest with an adviser.
6. Plan the trial. Choose a de-identified sample of your own data and agree the expected answers before any vendor sees it.
7. Send the same document to every vendor. Score answers against the requirements, not against the demo.

Worked example: an insurance broker’s email triage tool

Illustrative example: a regional New Zealand insurance broker with about 30 staff buying an AI tool to sort inbound client emails and draft replies. The business and notes are fictional.

Turn my rough notes into numbered vendor requirements for an AI email
triage tool. For each give: ID, one testable "must" or "should"
statement, how we will verify it, and the evidence the vendor supplies.
Group by: Job and accuracy, Data handling, Security and admin,
Commercial, Support and exit.
Do not add legal or regulatory requirements I have not mentioned; list
any you think I should check with an adviser. Flag vague notes.

Notes: sort client emails into claims, renewals, new business, other.
Draft replies for brokers to approve; never send on its own. Works with
our Microsoft 365 mailboxes. No training on our data. Want to know
where data goes; NZ or Australia preferred. Need to see what it did and
why. Price per mailbox? Caps? What happens if we leave?

Example output (illustrative, abridged):

R1  Must classify inbound client emails as Claims, Renewals, New
    business or Other. Verify: trial on 300 de-identified past emails,
    isolated from live systems, against our brokers' labels.
    Evidence: trial results by category, including errors.
R2  Must never send an email without a named broker's approval.
    Verify: confirm auto-send cannot be enabled, or is admin-locked off.
R5  Must not use our inputs, outputs or metadata to train or improve
    models, by default and under the contract.
R6  Must state where data is stored and processed and list all
    sub-processors. Should store data in New Zealand or Australia.
R7  Must notify us of any security incident affecting our data within
    [period to set].
R14 Must export our data, labels and settings in a standard format,
    and delete our data within [period to set], confirmed in writing.
Too vague to test: "see what it did and why". Suggested: "Must log
each classification with time, category and the approving broker."
Check with an adviser: any obligations specific to insurance broking.

Notice the placeholders. The assistant did not invent a breach notification period or a deletion deadline, and it did not invent insurance regulations. Those are your decisions.

Where AI vendor requirements go wrong

• Generic checklists. A hundred standard questions and none about your job. Start from your problem and measure.
• Untestable requirements. “User-friendly” and “accurate” cannot be scored. Every line needs a test and evidence.
• Vendor-written requirements. A shortlisted vendor’s template favours that vendor. Write your own first.
• No exit plan. MBIE’s checklist asks whether you can change providers and “whether its data is able to be easily migrated”. Settle export and deletion before signing.
• Certificates as proof. ISO/IEC 42001 certification shows an AI management system is in place, not that the product works on your emails.
• Invented regulations. AI drafting can add plausible legal requirements. Check every one.

Cross-check your list with our AI vendor evaluation checklist and AI procurement questions. For data location detail, see data residency for AI systems, and if no product fits, weigh custom AI versus off the shelf tools. Use the figures from creating an AI budget for the commercial section. For higher-risk purchases, our AI governance advisory and AI security reviews can review requirements and vendor answers independently.

Self-check

• Every requirement is numbered, testable and names its evidence.
• Accuracy will be tested on your own de-identified data, isolated from live systems.
• Data handling covers training use, location, sub-processors, breach notice and deletion.
• Security covers sign-in, roles, audit logs and human approval settings.
• Pricing, caps, support and exit terms are included.
• Every legal or regulatory reference has been checked by a person.
• The same document goes to every vendor.

Next lesson: Planning a safe AI rollout.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.