AI agent acceptance checklist.

Use this checklist at the end of a build and before every major change, such as a new model, a new tool or wider permissions. It applies to any AI system that takes actions: sending messages, updating records, raising orders or calling other systems. Five Eyes guidance co-authored by NCSC New Zealand says agentic AI should never get broad or unrestricted access, should be used only for low-risk, non-sensitive tasks, and needs human approval checkpoints for high-impact actions (NCSC).

Codes in brackets refer to the OWASP Top 10 for Agentic Applications for 2026 (ASI) and the OWASP Top 10 for LLM Applications 2026 (LLM).

The acceptance checklist

1. Scope and permissions

• The agent’s job, and what it must never do, is written down and signed by the owner.
• Each tool uses its own credentials with the least access the job needs (ASI03).
• Read and write access are separated; write access is limited to named records or folders.
• The agent cannot reach systems outside its scope, tested by trying.
• No passwords, keys or confidential notes sit in prompts or hidden context (LLM08).

2. Task performance

• A test set of at least 50 real cases, with known correct outcomes, has been run.
• The pass rate meets the threshold agreed before testing.
• High-impact cases were tested and scored separately.
• When unsure, the agent escalates instead of guessing.
• Results are consistent across three runs of the test set.

3. Human approval and escalation

• Payments, external messages, record changes and anything hard to undo need approval.
• The approver sees what the agent will do and why, before approving.
• If an approval times out, the agent takes no action.
• The escalation path to a person has been tested end to end.

4. Security

• Hidden instructions in emails, documents and web pages the agent reads do not change its actions (LLM01, ASI01).
• Tools cannot be misused for purposes outside the job, tested with deliberate attempts (ASI02).
• Outputs are validated before being passed to other systems (LLM10).
• Stored memory or context cannot be poisoned by one user to affect another (ASI06).
• The agent will not reveal its instructions or other users’ data when asked (LLM08).

5. Failure handling

• Retries are limited in number and time.
• Repeating an action after a retry does not create duplicates, such as a second order.
• A failure part-way through leaves records consistent, or is flagged for a person.
• An outage in a connected system is handled and reported.
• A failure in one step cannot cascade through other agents or systems (ASI08).

6. Logging and audit

• Every tool call is logged with its inputs, outputs and time.
• Each run can be traced to the case, user and approver involved.
• Logs containing personal information are protected and kept only as long as policy allows.
• A past run can be reconstructed from the logs.

7. Cost and limits

• Each run has a maximum number of steps and tokens (LLM06).
• A monthly spending cap and alert are set.
• Loops are detected and stopped.
• Cost per completed task was measured in testing and is within budget.

8. Operations and exit

• The off switch has been tested, and staff know how to use it.
• A named owner and a runbook exist.
• A model change requires re-running the test set before release.
• Support hours and the incident process are agreed.

Sign-off rules

How to run acceptance testing

1. Agree criteria before testing. Pass thresholds and must-pass items are signed first.
2. Build the test set from real work. Include edge cases and every high-impact action. Our guide to AI agent evaluations covers test design.
3. Test in a copy of live systems. Use sandbox accounts or demo organisations where your software provides them.
4. Attack it. Plant instructions in test emails and documents; see prompt injection testing.
5. Break its dependencies. Switch off a connected system mid-run and check what happens.
6. Record evidence and sign off. Fails go back to the builder with the failing case attached.

Filled-in example

Illustrative example: a Queenstown hospitality group tests an agent that reads supplier invoices, matches them to purchase orders and drafts bills in Xero for a person to approve, a version of supplier invoice processing.

Common mistakes

• Accepting on a demo. Demos show the happy path; acceptance tries to make the agent fail.
• Testing only what it should do. The “must not” tests matter more.
• Broad permissions for convenience. Read our guide to agent permissions and approvals before granting write access.
• No re-test after a model change. A new model is a new system.

Next step

Design the agent to pass these tests from the start with designing reliable AI agents. For an example of an internal agent where staff keep the decisions, see the property management agent, or talk to us about AI agent development.

Written by the Levy Street team in Wellington. Last updated 19 October 2026.