AI vendor evaluation checklist.

This checklist is for the stage after you have a shortlist. It works for buying a subscription tool, an industry product with AI features, or a supplier who will build something for you. Score each vendor on the same evidence, and keep the evidence: you will need it when the contract is negotiated and when someone asks later why you chose it.

How to run the evaluation

1. Write down the job. One paragraph on the workflow, volume and who reviews output. Our lesson on writing AI vendor requirements shows the format.
2. Prepare 20 to 30 real test cases. Include awkward ones. Remove or replace personal information unless the vendor is already approved for it.
3. Apply the knock-out rules first. A desk check against published terms removes vendors you could never use.
4. Run every remaining vendor on the same cases. Record how many outputs are usable after review, and how long review takes.
5. Collect written evidence. Send the AI procurement questions and attach the answers to this sheet.
6. Score, weight and decide. Use the scoring sheet below. Two people score independently, then agree.

The vendor evaluation checklist

1. Fit for the workflow

• Does the job described, end to end, without a workaround we would have to build.
• Works inside the systems we already use, or connects to them with a supported connector or API.
• Has the admin and approval steps our review process needs.
• Has reference customers of a similar size or sector we can call.

2. Accuracy on our own work

• Tested on our 20 to 30 cases, not the vendor’s demo data.
• Share of usable outputs after review recorded for each vendor.
• Review time per output recorded.
• Failure modes noted: what it gets wrong, and whether the errors are obvious or subtle.
• Shows sources or reasoning so a reviewer can check its claims.

3. Data use and privacy

• Written confirmation that inputs and outputs are not used for training on the plan we will buy.
• Retention period known, and deletion on request possible.
• Countries of storage and processing listed, including sub-processors.
• Acts only on our behalf for personal information, in its terms or a data processing agreement.
• Personal information is handled in a way our privacy officer has approved.

4. Security and access

• Single sign-on and multi-factor authentication supported.
• We can add and remove users ourselves, immediately.
• Role-based permissions limit who sees what.
• Audit logs show who did what, and we can export them.
• Holds or follows a recognised security framework, with evidence.

5. Integration and operations

• Connectors or APIs for our systems are documented and supported.
• Our other software’s terms allow the connection: for example, Xero’s developer terms prohibit using data from its API to train AI models (Xero).
• The vendor gives notice before changing or retiring the models it uses.
• Support hours suit New Zealand time zones.

6. Cost and commercial terms

• Price at our expected volume calculated, not just the entry price.
• Seat minimums, usage limits and overage charges understood.
• Currency and GST treatment confirmed.
• Price increase terms and notice period in writing.

7. Exit and continuity

• We can export our data, prompts and configuration in a usable format.
• Notice period and termination terms are acceptable.
• We know what happens to our data after cancellation.
• A credible fallback exists if the vendor fails or is acquired.
• The plan for running the workflow without the tool is written down.

Knock-out rules

Exclude a vendor, whatever its score, if any of these is true:

Scoring sheet

Score each group 0 to 3: 0 no evidence, 1 claimed by the vendor, 2 documented in terms or product documentation, 3 tested by us. Multiply by the weight, then add up. Maximum is 300.

Change the weights to suit your workflow, but set them before testing starts.

Filled-in example

Illustrative example: a 30-person Wellington law firm on Microsoft 365 compares three options for first drafts of client letters: Tool A (the assistant built into its office suite), Tool B (a standalone assistant on a team plan) and Tool C (a legal-specific product).

Tool C failed a knock-out: it could not say where data was processed. Tool A won on data and integration despite slightly weaker drafts, and the firm planned a six-week pilot.

Common mistakes

• Scoring the demo. Vendors demo their best cases. Only your cases count.
• Ignoring review time. A tool that is right 90% of the time but slow to check can lose to one right 80% of the time with sources shown.
• Comparing entry prices. Seat minimums and paying for unassigned seats change the real cost; ChatGPT Business, for example, charges for every purchased seat whether assigned or not (OpenAI).
• Assuming local processing. OpenAI’s Australian data residency covers storage at rest, not processing (OpenAI), and Anthropic’s API offers only global or US-only inference (Anthropic).

Next step

Take the winner into a time-boxed pilot scored with the AI pilot scorecard. If the choice is between buying and building, our comparison of custom AI and off-the-shelf tools and our AI model selection service cover the trade-offs.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.