# AI vendor evaluation checklist.

URL: https://levystreet.com/resources/ai-vendor-evaluation-checklist/
Updated: 2026-10-19

AI vendor evaluation checklist for NZ buyers: 32 checks in seven groups, knock-out rules and a weighted scoring sheet for comparing shortlisted AI tools.

## Short answer

An AI vendor evaluation checklist compares shortlisted AI tools or suppliers on fit, accuracy on your own work, data use, security, integration, cost and exit. This version has 32 checks in seven groups, knock-out rules that exclude a vendor whatever its score, and a weighted scoring sheet that rewards tested evidence over claims, written for New Zealand buyers.

- Score evidence, not claims: tested beats documented, documented beats promised.
- Knock-out rules exclude a vendor regardless of its total score.
- Run every vendor on the same 20 to 30 real examples.
- No major AI provider guaranteed New Zealand-only model processing as at October 2026.

This checklist is for the stage after you have a shortlist. It works for buying a subscription tool, an industry product with AI features, or a supplier who will build something for you. Score each vendor on the same evidence, and keep the evidence: you will need it when the contract is negotiated and when someone asks later why you chose it.

## How to run the evaluation


1. **Write down the job.** One paragraph on the workflow, volume and who reviews output. Our lesson on [writing AI vendor requirements](/learn/writing-ai-vendor-requirements/) shows the format.
2. **Prepare 20 to 30 real test cases.** Include awkward ones. Remove or replace personal information unless the vendor is already approved for it.
3. **Apply the knock-out rules first.** A desk check against published terms removes vendors you could never use.
4. **Run every remaining vendor on the same cases.** Record how many outputs are usable after review, and how long review takes.
5. **Collect written evidence.** Send the [AI procurement questions](/resources/ai-procurement-questions/) and attach the answers to this sheet.
6. **Score, weight and decide.** Use the scoring sheet below. Two people score independently, then agree.


## The vendor evaluation checklist

### 1. Fit for the workflow


- Does the job described, end to end, without a workaround we would have to build.
- Works inside the systems we already use, or connects to them with a supported connector or API.
- Has the admin and approval steps our review process needs.
- Has reference customers of a similar size or sector we can call.


### 2. Accuracy on our own work


- Tested on our 20 to 30 cases, not the vendor's demo data.
- Share of usable outputs after review recorded for each vendor.
- Review time per output recorded.
- Failure modes noted: what it gets wrong, and whether the errors are obvious or subtle.
- Shows sources or reasoning so a reviewer can check its claims.


### 3. Data use and privacy


- Written confirmation that inputs and outputs are not used for training on the plan we will buy.
- Retention period known, and deletion on request possible.
- Countries of storage and processing listed, including sub-processors.
- Acts only on our behalf for personal information, in its terms or a data processing agreement.
- Personal information is handled in a way our privacy officer has approved.


### 4. Security and access


- Single sign-on and multi-factor authentication supported.
- We can add and remove users ourselves, immediately.
- Role-based permissions limit who sees what.
- Audit logs show who did what, and we can export them.
- Holds or follows a recognised security framework, with evidence.


### 5. Integration and operations


- Connectors or APIs for our systems are documented and supported.
- Our other software's terms allow the connection: for example, Xero's developer terms prohibit using data from its API to train AI models ([Xero](https://developer.xero.com/faq/pricing-and-policy-updates)).
- The vendor gives notice before changing or retiring the models it uses.
- Support hours suit New Zealand time zones.


### 6. Cost and commercial terms


- Price at our expected volume calculated, not just the entry price.
- Seat minimums, usage limits and overage charges understood.
- Currency and GST treatment confirmed.
- Price increase terms and notice period in writing.


### 7. Exit and continuity


- We can export our data, prompts and configuration in a usable format.
- Notice period and termination terms are acceptable.
- We know what happens to our data after cancellation.
- A credible fallback exists if the vendor fails or is acquired.
- The plan for running the workflow without the tool is written down.


## Knock-out rules

Exclude a vendor, whatever its score, if any of these is true:

| Knock-out | Why it matters |
|---|---|
| Trains on our inputs on the plan we can buy, with no opt-out | The Privacy Commissioner advises against entering personal or confidential information unless the provider has confirmed it is not retained or disclosed ([OPC](https://www.privacy.org.nz/resources-and-learning/a-z-topics/ai/generative-artificial-intelligence/)) |
| Cannot tell us where data is processed | You cannot assess offshore privacy risk |
| We cannot remove a user's access ourselves | Departing staff keep access to work data |
| No way to export our data | You are locked in from day one |

## Scoring sheet

Score each group 0 to 3: **0** no evidence, **1** claimed by the vendor, **2** documented in terms or product documentation, **3** tested by us. Multiply by the weight, then add up. Maximum is 300.

| Group | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Fit for the workflow | 20 | | | |
| Accuracy on our work | 30 | | | |
| Data use and privacy | 15 | | | |
| Security and access | 10 | | | |
| Integration and operations | 10 | | | |
| Cost and commercial | 10 | | | |
| Exit and continuity | 5 | | | |
| **Weighted total (of 300)** | | | | |

Change the weights to suit your workflow, but set them before testing starts.

## Filled-in example

Illustrative example: a 30-person Wellington law firm on Microsoft 365 compares three options for first drafts of client letters: Tool A (the assistant built into its office suite), Tool B (a standalone assistant on a team plan) and Tool C (a legal-specific product).

| Group | Weight | Tool A | Tool B | Tool C |
|---|---|---|---|---|
| Fit | 20 | 2 (40) | 2 (40) | 3 (60) |
| Accuracy on 25 test letters | 30 | 2 (60) | 3 (90) | 3 (90) |
| Data use and privacy | 15 | 3 (45) | 2 (30) | 1 (15) |
| Security and access | 10 | 3 (30) | 2 (20) | 2 (20) |
| Integration | 10 | 3 (30) | 1 (10) | 2 (20) |
| Cost | 10 | 3 (30) | 2 (20) | 1 (10) |
| Exit | 5 | 2 (10) | 2 (10) | 1 (5) |
| **Total** | | **245** | **220** | **220** |

Tool C failed a knock-out: it could not say where data was processed. Tool A won on data and integration despite slightly weaker drafts, and the firm planned a six-week pilot.

## Common mistakes

- **Scoring the demo.** Vendors demo their best cases. Only your cases count.
- **Ignoring review time.** A tool that is right 90% of the time but slow to check can lose to one right 80% of the time with sources shown.
- **Comparing entry prices.** Seat minimums and paying for unassigned seats change the real cost; ChatGPT Business, for example, charges for every purchased seat whether assigned or not ([OpenAI](https://help.openai.com/en/articles/8792536-managing-billing-and-seats-in-chatgpt-business)).
- **Assuming local processing.** OpenAI's Australian data residency covers storage at rest, not processing ([OpenAI](https://help.openai.com/en/articles/9903489-data-residency-and-inference-residency-for-chatgpt)), and Anthropic's API offers only global or US-only inference ([Anthropic](https://platform.claude.com/docs/en/about-claude/pricing)).

## Next step

Take the winner into a time-boxed pilot scored with the [AI pilot scorecard](/resources/ai-pilot-scorecard/). If the choice is between buying and building, our comparison of [custom AI and off-the-shelf tools](/compare/custom-ai-vs-off-the-shelf/) and our [AI model selection](/services/ai-model-selection/) service cover the trade-offs.

## Questions people ask

### How do you evaluate an AI vendor?

Shortlist two or three, then run each on the same set of real examples from your business and score the results. Alongside the test, collect written evidence on data use, where data is processed, security, admin controls, pricing at your expected volume and how you would leave. Exclude any vendor that fails a knock-out rule, then compare weighted scores for the rest.

### What questions should I ask about how an AI vendor uses my data?

Ask whether your inputs and outputs are used to train or improve their models, on the exact plan you will buy; how long they are kept; which countries they are stored and processed in; which sub-processors see them; and whether they act only on your behalf. Get the answers in the contract or published terms, not in a sales email.

### Can an AI vendor process our data in New Zealand?

As at October 2026, no major provider guaranteed model processing inside New Zealand only. The closest option is Claude through Amazon Bedrock's Australian profile, which keeps inference within Australia and New Zealand. OpenAI's Australian region stores data but does not process it there, and Anthropic's own API offers global or US-only inference. Record where processing happens and decide whether that suits your data.

### How many vendors should we evaluate?

Two or three. Evaluating more takes longer than it saves, because the expensive part is preparing real test cases and checking outputs. Use a quick desk check against the knock-out rules to cut a long list down, then spend the effort on testing the shortlist properly against your own work.


## Sources

- [Artificial intelligence for small business](https://www.ncsc.govt.nz/assets/guidance/Documents/Artificial-Intelligence-for-Small-Business_D7.pdf)
- [Generative Artificial Intelligence](https://www.privacy.org.nz/resources-and-learning/a-z-topics/ai/generative-artificial-intelligence/)
- [Data residency and inference residency for ChatGPT](https://help.openai.com/en/articles/9903489-data-residency-and-inference-residency-for-chatgpt)
- [Claude API pricing](https://platform.claude.com/docs/en/about-claude/pricing)
- [Run generative AI inference with Amazon Bedrock in Asia Pacific (New Zealand)](https://aws.amazon.com/blogs/machine-learning/run-generative-ai-inference-with-amazon-bedrock-in-asia-pacific-new-zealand/)
- [Pricing and policy updates](https://developer.xero.com/faq/pricing-and-policy-updates)
- [Managing billing and seats in ChatGPT Business](https://help.openai.com/en/articles/8792536-managing-billing-and-seats-in-chatgpt-business)

Published by Levy Street (https://levystreet.com), Wellington, New Zealand.